Skip to content

hdfs: migrate to fsspec - #6604

Merged
efiop merged 2 commits into
treeverse:masterfrom
isidentical:pyarrowfs-migration
Oct 6, 2021
Merged

hdfs: migrate to fsspec#6604
efiop merged 2 commits into
treeverse:masterfrom
isidentical:pyarrowfs-migration

Conversation

@isidentical

Copy link
Copy Markdown
Contributor

Resolves #6582, waiting fsspec/filesystem_spec#754

@isidentical
isidentical requested a review from a team as a code owner September 13, 2021 17:45
@isidentical
isidentical requested a review from pared September 13, 2021 17:45
@isidentical isidentical changed the title hdfs: migrate to fsspec [WIP] hdfs: migrate to fsspec Sep 13, 2021
@isidentical
isidentical removed the request for review from pared September 13, 2021 17:46
@isidentical
isidentical marked this pull request as draft September 13, 2021 17:46
@isidentical

Copy link
Copy Markdown
Contributor Author

This is awaiting a new fsspec release

@isidentical isidentical changed the title [WIP] hdfs: migrate to fsspec hdfs: migrate to fsspec Oct 2, 2021
@isidentical
isidentical marked this pull request as ready for review October 2, 2021 17:16
@isidentical
isidentical force-pushed the pyarrowfs-migration branch 5 times, most recently from db55554 to f581a73 Compare October 4, 2021 08:37

@pared pared left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far, Windows tests are failing.

@isidentical
isidentical force-pushed the pyarrowfs-migration branch 10 times, most recently from ccae888 to 6970f89 Compare October 6, 2021 08:20
@skshetry

skshetry commented Oct 6, 2021

Copy link
Copy Markdown
Collaborator

@isidentical, the failures are due to a new release in dvclive. I have pinned it to 0.3.0 for now, please rebase.

@isidentical

Copy link
Copy Markdown
Contributor Author

@mergify rebase

@mergify

mergify Bot commented Oct 6, 2021

Copy link
Copy Markdown
Contributor

Command rebase: success

Branch has been successfully rebased

Hey, I reacted but my real name is @Mergifyio

@skshetry
skshetry force-pushed the pyarrowfs-migration branch from 6970f89 to 20791f9 Compare October 6, 2021 10:34
@efiop
efiop merged commit 054feae into treeverse:master Oct 6, 2021
@efiop efiop added the fs: hdfs Related to the HDFS filesystem label Oct 6, 2021
skshetry added a commit to endremborza/dvc that referenced this pull request Oct 8, 2021
* master: (47 commits)
  make dependabot updates work (treeverse#6760)
  move defaults to the class level (treeverse#6762)
  build(deps): Bump dvclive from 0.3.0 to 0.4.0 (treeverse#6753)
  move requirements to setup.cfg (treeverse#6758)
  gha: cancel previous workflows (treeverse#6756)
  pyinstaller: hooks: remove webhdfs
  webhdfs: migrate to fsspec (treeverse#6662)
  gha: benchmarks: fix pytest flags (treeverse#6749)
  hdfs: migrate to fsspec (treeverse#6604)
  Pin dvclive to 0.3.0
  command: Should not consume 'print' to output default remote info (treeverse#6650)
  command: output prettify json in tty (treeverse#6743)
  gha: run benchmarks on PRs (treeverse#6733)
  Run on Python3.10 (treeverse#6745)
  Fix dvclive running subdir (treeverse#6740)
  exp init: only ask for that are not provided in an interactive mode (treeverse#6739)
  move exp init logic to dvc.repo.experiments.init (treeverse#6738)
  build(deps): Bump jaraco-windows from 5.6.0 to 5.7.0 (treeverse#6735)
  build(deps): Bump pytest-cov from 2.12.1 to 3.0.0 (treeverse#6736)
  Bump typing_extensions to >=3.7.4 (treeverse#6731) (treeverse#6732)
  ...
@efiop efiop added the enhancement Enhances DVC label Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhances DVC fs: hdfs Related to the HDFS filesystem

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hdfs: migrate to fsspec/pyarrow.fs

4 participants