Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-27435: Unification of file-like datastores #426

Merged
merged 19 commits into from
Nov 11, 2020
Merged

DM-27435: Unification of file-like datastores #426

merged 19 commits into from
Nov 11, 2020

Commits on Nov 11, 2020

  1. Install webdav packages so we can run tests

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    5a4e828 View commit details
    Browse the repository at this point in the history

  2. Fix ResourceWarning in open

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    00c82b9 View commit details
    Browse the repository at this point in the history

  3. Add new dependency on deprecated package

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    952cf6b View commit details
    Browse the repository at this point in the history

  4. Unify S3 and WebDAV datastores 

    They are now deprecated and replaced by RemoteFileDatastore
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    36b75b6 View commit details
    Browse the repository at this point in the history

  5. mypy does not like usage of FutureWarning in deprecated so use Deprec… 

    …ationWarning
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    e284936 View commit details
    Browse the repository at this point in the history

  6. Unify __init__ for all file-like datastores

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    91f2488 View commit details
    Browse the repository at this point in the history

  7. Add ButlerURI.isTemporary property 

    This lets a URI from a temporary file be marked as such so that
users can make decisions (for example when deciding whether
certain transfer modes make sense).
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    a5f65c8 View commit details
    Browse the repository at this point in the history

  8. Add an isLocal property to simplify a check for local file 

    This will save people from doing a check on the URI scheme
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    c75e201 View commit details
    Browse the repository at this point in the history

  9. Rewrite ButlerURI.as_local as a context manager 

    This allows temporary files to be deleted automatically.
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    1d06a71 View commit details
    Browse the repository at this point in the history

  10. Fix file Formatter fromBytes to handle missing component 

    The file reader had this logic but the bytes handler did not.
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    4b69996 View commit details
    Browse the repository at this point in the history

  11. Add method to Formatter to indicate if bytes can be processed 

    The implementation is not perfect since it relies on a method
raising NotImplementedError since at the moment there is no
class property to simply declare bytes support.
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    4550b72 View commit details
    Browse the repository at this point in the history

  12. Unify _read_artifact_into_memory for POSIX and Remote

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    5a33b42 View commit details
    Browse the repository at this point in the history

  13. Move write artifact into unified FileLikeDatastore

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    5f13f39 View commit details
    Browse the repository at this point in the history

  14. Switch all internals and defaults to FileLikeDatastore 

    * PosixDatastore and RemoteFileDatastore now issued
  deprecation warnings.
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    44a8cdc View commit details
    Browse the repository at this point in the history

  15. Remove warning if dataset exists in datastore

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    e81e6ac View commit details
    Browse the repository at this point in the history

  16. Improve the code that temporarily changes the formatter location 

    Now has a context manager rather than directly changing the
internal property of the formatter.
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    2b3d9f1 View commit details
    Browse the repository at this point in the history

  17. Rename FileLikeDatastore to FileDatastore

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    dd10fda View commit details
    Browse the repository at this point in the history

  18. Remove RemoteFileDatastore 

    Nothing ever used it directly so it can be safely removed without
a deprecation period.
    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    446edc4 View commit details
    Browse the repository at this point in the history

  19. Remove comment that has not been relevant for a long time

    @timj
    timj committed Nov 11, 2020
    Configuration menu
    Copy the full SHA
    19300f8 View commit details
    Browse the repository at this point in the history