Deprecate Object API

[willscott]

The Object API is a not-well documented portion of the command line API that allows for working with nodes of data serialized to a range of codes. The DAG API was meant to deprecate it, and has existed since 2017.

• Is there anything being done by Object API that is not possible otherwise?
• Is there any compelling reason to keep it around?

[willscott]

cc @vmx who asked to follow this discussion to make sure the JS implementation side ends up in the same place.

[aschmahmann]

A cursory overview of the commands that ipfs object utilizes that are not available via ipfs dag are:

• ipfs object links (Update: replaceable via ipfs refs <cid>)
• Some DagPb or UnixFS utilities
  • The ipfs object patch commands (i.e. mutating a DagPb node)
  • ipfs object diff which is pretty nice for diff-ing UnixFS directories, and perhaps useful to people who have built IPLD structures out of DagPb
  • ipfs object new unixfs-dir for creating an empty unixfs directory
  • ipfs object data for getting the raw bytes out of the data field of a DagPb node

Reasons to keep the API around:

• Functionality not available elsewhere
• Backwards compatibility

We could likely take the useful bits of this API and add them to the DAG API (or to ipfs files) and once that's done start deprecating the Object API. Likely we'd still leave in the commands with a notice that they're deprecated unless it's unreasonable since we try not to break CLI scripts if it's not too rough to avoid. However, IMO we could deprecate and then remove the Object part of the CoreAPI.

[mikeal]

We’ve been saying the object API is “the old thing you shouldn’t use” since before I joined, so about 3 years now.

IMO, these features aren’t actually useful, nor is the dag API very useful. It’s really hacky to try and interact with graph nodes with this CLI so I question its utility. If you felt you must interact with a CLI to manipulate graph nodes it would be easier to use CLI tools for CBOR/JSON and pipe them to the Block CLI.

This is even more true for this as a remote or library API. You can write a much better interface doing the codec work in the client and reading/writing to the Block API. We’ve been telling people to do this instead of using the dag API for at least a year.

If we’re going to be ambitious and take on new work we also need to jettison features we don’t think are worth maintaining and, IMO, these both qualify. I’m not aware of any major users that depend on them and I don’t know of any use cases they serve that you couldn’t find a better solution to using the Block API/CLI which is much simpler and easier to maintain.

[aschmahmann]

We’ve been saying the object API is “the old thing you shouldn’t use” since before I joined, so about 3 years now.

Maybe, but saying "you shouldn't use" something doesn't really hold any power until an alternative arises. Since the originally proposed replacement (i.e. the DAG API) never matured enough to fully replace the Object API it hasn't been deprecated yet.

I don’t know of any use cases they serve that you couldn’t find a better solution to using the Block API/CLI which is much simpler and easier to maintain.

• ipfs object diff, as mentioned above, does diffs of UnixFS directories without making you download the whole thing. If you have a better tool though I'd be happy to use it in https://github.com/ipfs/distributions
• ipfs patch is likely replaceable with some MFS tooling for UnixFS directories (e.g. allowing ipfs files cp to work outside the go-ipfs MFS root), but IMO using the Block API here would not be fun

So sure, maybe just moving the useful commands into ipfs files is sufficient for most users of ipfs object since we've been pushing people away from DagPb for a while. However, we should still support DagPb users and make sure they have alternative tools available if the ones they were relying on become deprecated.

You can write a much better interface doing the codec work in the client and reading/writing to the Block API. We’ve been telling people to do this instead of using the dag API for at least a year.

That make sense since anyone using DAG CBOR doesn't have any way of doing mutations of graphs in go-ipfs without manually updating the nodes in the DAG that are changed and all of their parents. However, it seems doubtful to me that this is the best way of doing things.

Perhaps I'm wrong and working with bytes is the best way forward. However, I suspect people are going to want to manipulate DAGs with go-ipld-prime and that it is going to need (or already has?) tooling around mutating DAGs. Once we see how people want to manipulate go-ipld-prime DAGs where the bytes are stored in a go-ipfs node we'll likely learn more about what patterns people are looking for.

[achingbrain]

People use the object API to do things like manipulate non-MFS UnixFS DAGs (e.g. add files to directories, create sub-directories, etc). There isn't a convenient way to do that with the current API without getting deep into the DagPb weeds.

I would make MFS-style operations (e.g. mkdir, rm, etc) work on regular UnixFS DAGs and each operation would return a CID representing the root of the affected DAG.

We should then go a step further and move all of the MFS operations to the root IPFS object and have them operate on both IPFS and MFS paths.

E.g.

const cid1 = ipfs.add({ path: '' })
const cid2 = ipfs.mkdir(`${cid1}/bar`)
const cid4 = ipfs.cp(cid3, `${cid2}/baz`)
// etc

You can use the DAG API to do more complex manipulations, but if you're down there you're probably not using UnixFS/DagPb any more so you should be ready for the full IPLD experience.

[Gozala]

Ideally I think we would:

1. Release a small library that provides object compatible API.
2. Facade existing API with deprecation warnings saying API will be gone in next release and delegate calls to that library.
3. In next release remove that library.

We don’t have to maintain that library, pass a maintenance to teams who don’t want to leave it behind.

[TheDiscordian]

• ipfs object links (Update: replaceable via ipfs refs <cid>)

Just my $0.02 as a developer new to IPFS using the HTTP API. While refs is cool, I like ipfs object links as it also outputs file/directory names. Having that on the CLI is nice as well.

I like object stat because it's fast, and I can use it to get the size to ensure directories are below 1 MiB, until Experimental.ShardingEnabled gets out of experimental at least.

[warpfork]

Thanks for that feedback @TheDiscordian ! That's very concrete and useful.

Those observations are interesting. It sounds like a lot of the operations described there semiotically asking about unixfs traits, and so ideally they are things we'd like to serve through an API that's more explicit in that it's doing unixfs things. I know the comment is about things that can be done with the object API today and are useful, but they're also probably very meaningful as feature requests for the APIs that should be replacing it.

[aschmahmann]

While refs is cool, I like ipfs object links as it also outputs file/directory names. Having that on the CLI is nice as well.

What is ipfs object links doing for you that ipfs ls isn't? There are so many CLI commands at the moment, and while most have their use I think what sticks out about the object API is that it's basically the DagPB API, not quite DAG agnostic and not quite UnixFS specific. So it seems the question is which parts are people using/why and then where to move those pieces.

I like object stat because it's fast, and I can use it to get the size to ensure directories are below 1 MiB

Can you be a little more precise here, because I think I'm missing something. There's ipfs block stat <cid> which will tell you the size of the block itself (e.g. the directory assuming no sharding), there's ipfs dag stat <cid> which will give you the cumulative size, and there's ipfs files stat /ipfs/<cid> which has more UnixFS specific information. Is there something about ipfs object stat that's missing from the others?

[TheDiscordian]

What is ipfs object links doing for you that ipfs ls isn't?

Nothing, ipfs ls works great, thanks for pointing me to it! I didn't even think of it, because when I think of ls I jump to files ls for MFS. My bad :).

Can you be a little more precise here, because I think I'm missing something. There's ipfs block stat <cid> which will tell you the size of the block itself (e.g. the directory assuming no sharding), there's ipfs dag stat <cid> which will give you the cumulative size, and there's ipfs files stat /ipfs/<cid> which has more UnixFS specific information. Is there something about ipfs object stat that's missing from the others?

So I was alerted to the directory size issue as a possible reason for #7916 before I opened the issue in a chat. Being somewhat unclear on what statistic I need, I explored into commands and found ipfs object stat which seemed to do what I want, but now I'm not sure. The functionality I'm looking for is ensuring the directory is under 1MiB, so it doesn't break (referring to this issue here, can't seem to find a Github one: https://discuss.ipfs.io/t/advice-for-starting-a-new-ipfs-cluster/10178). I'm actually still somewhat unclear on how to find this information:

Command: ipfs object stat /ipns/portal.thedisco.zone/osrs/w

Output:

NumLinks:       19317
BlockSize:      1136307
LinksSize:      1136305
DataSize:       2
CumulativeSize: 2089601752

I thought I'd need block size, assuming that's in bytes, that's reporting 1.08MiB, which is over the limit. Unless I should be looking at NumLinks?

block stat gives me the BlockSize, same info, and you're correct, the info I thought I needed. Perhaps I'm misunderstanding the 1MiB limit altogether?

dag stat gives cumulative, yes, but object stat immediately gives me a result, where dag stat takes several seconds (though it nicely updates me while it does so).

Command: ipfs files stat /ipfs-sync/DiscoNet/osrs/w (for context, it's the same as /ipns/portal.thedisco.zone/osrs/w)

Output:

QmcDdHgrPuAjoNWhkneSDYzZnDU1QQ4PYNPFKNFG3Uc2Z2
Size: 0
CumulativeSize: 2089601752
ChildBlocks: 19317
Type: directory

Which doesn't seem to contain information I'd typically be looking for (except CID, and maybe the type).

So to answer your question somewhat, I'd have to say "I don't know, but maybe not". As I'm struggling to even find which statistic I need. There certainly is a lot of overlap though, and the only statistics I didn't see repeated were: NumLinks (ChildBlocks of files stat actually has this value if it's added to MFS), LinksSize, DataSize, and the same CumulativeSize value (I'm assuming object stat is less accurate, as it's a lot faster, but it does report a different value than dag stat, though files stat reports the same value, if it's in the MFS).

Extra note: Some option to display SI units or something would be awesome. I can't always tell if I'm looking at bytes, or block counts, or something else.

[lidel]

I made some changes to ipfs --help to prioritize dag and files APIs over object in #8010
Both CLI and HTTP API docs are generated from those strings, so this is a low-hanging fruit to get people to use object less.

[lidel]

FYSA I opened a PR which makes the deprecation VERY explicit and points users at modern replacements: #8098
In real use cases the functional difference should be minimal, and any regression should be solved by improving dag block and files commands.

[tx0c]

is there a 1-to-1 migrate guide on how to update existing ipfs object commands? the DAG api looks too low level, can't find how can ipfs object patch add-link ... work with DAG API?