Skip to content

Conversation

@skshetry
Copy link
Collaborator

No description provided.

@skshetry
Copy link
Collaborator Author

It's unclear to me why we are even using dvc-http and dvc-objects here.

@skshetry
Copy link
Collaborator Author

skshetry commented Dec 13, 2023

I'm running with this to unblock treeverse/dvc#10158.

@skshetry skshetry merged commit 6b3ef53 into main Dec 13, 2023
@skshetry skshetry deleted the replace-wrap-and-coro branch December 13, 2023 10:56
@pmrowla
Copy link
Contributor

pmrowla commented Dec 13, 2023

It's unclear to me why we are even using dvc-http and dvc-objects here.

It's so we can reuse the logic for batched transfers and the aiohttp large download timeouts. I didn't see the point in copy/pasting everything from dvc-http and dvc-objects generic transfer into scmrepo

@skshetry
Copy link
Collaborator Author

skshetry commented Dec 14, 2023

dvc-http is a dvc's plugin. I have fixed the immediate issue, but we are creating a circular dependency issue here with this.

Maybe we can add support for batched transfers for fsspec filesystems in dvc-objects?
The aiohttp large download timeouts fixes are small that we can copy it here as well (or upstream it to fsspec).

Ideally, the fix would be to not have our own filesystem wrappers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants