objects: migrate remote push/pull to objects.transfer

[pmrowla]

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

• adds objects.transfer for transferring objects between two ODBs
  • old remote._process functionality has been migrated into objects.transfer
  • all transfers now use objects.save as the underlying file transfer method (rather than directly calling fs.upload or fs.download)
• adds objects.status for comparing object status between two ODBs
  • old remote.status functionality has been migrated to objects.status.compare_status
• old remote.index functionality has been migrated into objects.db.index
  • ODB index sits in its own layer separate from the actual ODB
  • index code has been migrated to diskcache instead of direct sqlite usage
  • legacy (now unused) sqlite code has been removed from dvc.state
• push/pull/fetch now use unified objects.transfer
• import now uses unified objects.transfer
• dvc.remote is now removed

[pmrowla]

Still need to run dvc-bench tests

[pmrowla]

Had discussion with @efiop regarding potential follow ups to this PR:

• objects.transfer should operate on object IDs rather than objects (since they already essentially treated as naive object IDs anyways)
  • for handling trees, the tree contents can be lazily loaded during transfer as needed instead of passing loaded tree objects into transfer
• eventually objects.save usage should be replaced by transferring from staging into the destination ODB
  • all staged files/dirs should have actual entries in the staging ODB, as opposed to the current behavior where we only "stage" trees
  • staging/staged objects should potentially work like a reference filesystem
  • loading the object ID from the staging ODB would then open/read the file data from referenced location
  • i.e. git imports could then be loaded (or transferred) from the staging ODB like any other object, and the actual file data would be read from the git/scm fs when necessary

[efiop]

Btw, discussed with @isidentical that for some filesystems like (hdfs and future ssh) upload_fobj down below will no longer be atomic, so we might need to use a temporary path here and then just rename into place. (there is an option to wrap fs calls to make them atomic but that is error prone and ugly).

[pmrowla]

Seems to me like this might still need to be handled at the fs level, uploading to a temp path and renaming at the ODB level won't work for all of our filesystems (HTTP doesn't support move/rename)

[efiop]

@pmrowla That atomicity is not something that fs should care about when uploading/downloading, this is an odb-level behavior.

HTTP doesn't support move/rename

Are operations already atomic there? Or it just doesn't support rename at all anywhere?

[pmrowla]

In the HTTP case, it's atomic since the full POST/PUT request wouldn't be completed so the server should drop whatever was partially uploaded. And yeah, we don't support rename/move/copy at all since there's no HTTP method for that operation (unless you're using an extension built on top of HTTP like webdav)

It seems to me that both _upload and _upload_fobj should work the same way, and should both guarantee atomicity at the fs level - like how in localfs we do the explicit upload to tempfile and rename for both _upload and _upload_fobj

[efiop]

@pmrowla Thanks for clarifying!

_upload_fobj is temporary until fsspec migration is complete and we can use put/get[_file] directly.

fs atomicity is unlikely to be guaranteed by all filesystems and might actually be undesirable in some use cases outside dvc (e.g. you might want to upload as much of a file as you can, or you might not care about atomicity so you might not want to waste an API call for rename), so it seems like it could be more robust if we do that in our odb layer (or fs wrapper after all?) for now.

Clearly, it seems like it would be useful to have the knowledge about whether or not particular fs operations are atomic so that we could waste the least api calls possible, so maybe our fsspec_wrapper is indeed a pretty good place for it for now, similar how, IIRC, in C libraries you have atomic_* functions, we could have something like put_file and atomic_put_file or atomic=True or something. Maybe this could be useful for fsspec in general as well, not quite sure right now 🤔

[efiop]

Doesn't look like this PR is changing the old behaviour, so probably not worth blocking it because of it, but we'll def need to keep this in mind for the followups.

[pmrowla]

odb passed into status will not always be where we want to load dir trees from (we want to load from local cache and not stream from the remote when possible)

For now this is just a performance optimization to keep parity with existing behavior, but the need for this in general will be more obvious once the full oid migration is done

[pmrowla]

On my machine the dvc bench results for the current PR are comparable to master, as noted in #6308 (comment) there will be a follow up PR that changes our object collection and status/transfer to use object IDs (hash infos) instead of objects