Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-27435: Unification of file-like datastores #426

Merged
merged 19 commits into from Nov 11, 2020
Merged

DM-27435: Unification of file-like datastores #426

merged 19 commits into from Nov 11, 2020

Conversation

timj
Copy link
Member

@timj timj commented Nov 5, 2020

No description provided.

@timj timj force-pushed the tickets/DM-27435 branch 3 times, most recently from f855017 to 22d5443 Compare November 6, 2020 22:29
@timj timj marked this pull request as ready for review November 6, 2020 22:41
Copy link
Member

@TallJimbo TallJimbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I think all of my comments are on things you're already aware could use some cleanup, and none of them are critical for this ticket.

# There will be problems with the bytes we are supplying so ignore
pass
return True

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the long-term plan here to implement this in subclasses directly, and then make it abstract?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I haven't worked out the best approach yet but there is a Formatter cleanup ticket. I wanted an API here though so that I could at least ask the formatter about it even if the implementation was a bit odd. I had hoped to get away with always using the presence of a method to declare it but it seems like I'm going to have to use the approach of a class property and implementation method.

with uri.as_local() as local_uri:
# Have to update the Location associated with the formatter
# because formatter.read does not allow an override.
# This could be improved.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, this is indeed something I'd love to see fixed (but not on this ticket).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The FileDescriptor class was one of the first classes that Pim wrote and I'm not really convinced I need it. Again I can deal with that on the formatter cleanup.

python/lsst/daf/butler/datastores/fileLikeDatastore.py Outdated Show resolved Hide resolved
python/lsst/daf/butler/datastores/fileLikeDatastore.py Outdated Show resolved Hide resolved
@timj timj force-pushed the tickets/DM-27435 branch 3 times, most recently from 7e9a9f1 to 5fc9708 Compare November 11, 2020 00:00
They are now deprecated and replaced by RemoteFileDatastore
This lets a URI from a temporary file be marked as such so that
users can make decisions (for example when deciding whether
certain transfer modes make sense).
This will save people from doing a check on the URI scheme
This allows temporary files to be deleted automatically.
The file reader had this logic but the bytes handler did not.
The implementation is not perfect since it relies on a method
raising NotImplementedError since at the moment there is no
class property to simply declare bytes support.
* PosixDatastore and RemoteFileDatastore now issued
  deprecation warnings.
Now has a context manager rather than directly changing the
internal property of the formatter.
Nothing ever used it directly so it can be safely removed without
a deprecation period.
@timj timj merged commit 520f072 into master Nov 11, 2020
@timj timj deleted the tickets/DM-27435 branch November 11, 2020 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants