Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add git to filesystem source 301 #954

Draft
wants to merge 8 commits into
base: devel
Choose a base branch
from

Commits on Feb 15, 2024

  1. Add gitpythonfs fsspec implementation

    - Intended to be used in filesystem verified Source.
    -read-only
    - due to depedency of gitpython on git binaries, gitpythonfs should only be conditionally imported, for example by registering the module using  `register_implementation_in_fsspec()`
    deanja committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    b04d563 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    43c6180 View commit details
    Browse the repository at this point in the history
  3. Urls do not require a netloc (in rfc1808)

    - See https://www.rfc-editor.org/rfc/rfc1808.html#section-2.1 . So, for example, gitpythonfs:// is valid and works in fsspec.
    - However in bucket-based systems, the bucket name is where the netloc would be - eg s3://bucket_name.  But luckily s3fs (and probably az, gcs etc) already gives helpful error messages if no bucket name given, wrong bucket name given etc.
    - As a ToDo, this rule could test only those protocols whose fsspec implementation needs netloc, rather than excluding a few protocols as is done here. But we don't know which protocols this rule was initially added for.
    deanja committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    c16a9dc View commit details
    Browse the repository at this point in the history
  4. Use 'mtime' as default input for modification_date

    -reduces need to modify code every time a new fsspec implementation is added
    - `mtime` is idiomatic in *nix file systems.
    deanja committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    ea0557f View commit details
    Browse the repository at this point in the history
  5. add default mtime

    deanja committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    87ec256 View commit details
    Browse the repository at this point in the history
  6. move gitpythonfs

    deanja committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    5d644fc View commit details
    Browse the repository at this point in the history
  7. move dynamic registration

    deanja committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    62790c1 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. use distinct filesystem instances in pipeline threads

    - clearer naming in FileItemDict
    deanja committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    b473fb5 View commit details
    Browse the repository at this point in the history