Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-24475: Rewrite of Butler URI and major clean up of S3 usage #336

Merged
merged 45 commits into from
Jul 30, 2020
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
f02ff92
Add more transfer modes to posix export
timj Jul 21, 2020
1cd2aff
Move ButlerURI to separate file
timj Jul 21, 2020
54208da
Refactor ButlerURI to start creating subclasses
timj Jul 21, 2020
8287312
Refactor the URI fix up code for subclasses
timj Jul 22, 2020
f926cb7
Simply schemeless fix up code
timj Jul 22, 2020
0641357
Add concrete methods for transfer and exists of URI resources
timj Jul 23, 2020
b2ce3f8
Change ButlerURI.dirname so that it does not force absolute path
timj Jul 23, 2020
b4c8cce
Remove abstract method
timj Jul 23, 2020
12a4ddd
Add ButlerURI.join
timj Jul 23, 2020
a8f873e
Switch posixDatastore export to using ButlerURI.transfer_from
timj Jul 23, 2020
f2c0f62
Add auto transfer mode for ButlerURI
timj Jul 23, 2020
dfb206e
Move datastore.export to file base class and enable S3 test
timj Jul 23, 2020
446f779
Add ButlerURI.read method
timj Jul 23, 2020
640f65d
Add ButlerURI.isabs
timj Jul 24, 2020
ff38951
Add some checks for S3 exceptions
timj Jul 24, 2020
ad31c1c
Fix join dirLike property
timj Jul 24, 2020
dcbc0cd
Add pkg resource URI support
timj Jul 24, 2020
bbff6db
Replace S3 and Resource calls with ButlerURI
timj Jul 24, 2020
c2fe7ab
Fix some relative imports of ButlerURI now that it moved
timj Jul 24, 2020
4ca5e3a
Add ButlerURI.write() for writing bytes to the resource
timj Jul 24, 2020
86f9917
For local files check if URI is a directory
timj Jul 24, 2020
a20c8a4
Reimplement Config.dumpToUri to use ButlerURI.write
timj Jul 24, 2020
6164272
Clean up more of the dirLike logic
timj Jul 24, 2020
4c9227e
Add ButlerURI.mkdir
timj Jul 24, 2020
38d9e32
Move S3/file directory creation code out of makeRepo to ButlerURI
timj Jul 24, 2020
8894565
Add ButlerURI.relative_to for determining subpaths
timj Jul 24, 2020
f9bef58
Rewrite S3 ingest to use ButlerURI
timj Jul 24, 2020
4602a74
Fix ButlerURI.relative_to for schemeless vs file URIs
timj Jul 24, 2020
94190dd
Use uri relative_to method in posix datastore
timj Jul 24, 2020
53acda9
Add debug logging for rollback undo function
timj Jul 25, 2020
15a4ca2
Add some more commentary to ingest test
timj Jul 25, 2020
d77ca4e
Fix mistaken realpath read
timj Jul 25, 2020
209137b
Use local tempdir to work around relsymlink problem
timj Jul 25, 2020
c456d35
Add transaction and move support to ButlerURI
timj Jul 25, 2020
e0b62c3
Use ButlerURI.transfer_from for ingest in posix datastore
timj Jul 25, 2020
e6a2a62
Add comment on full path for S3
timj Jul 25, 2020
0f5e3ca
Improve logging on rollback
timj Jul 25, 2020
c9181fb
Use the new ButlerURI.join() method rather than hand rolling
timj Jul 25, 2020
f6b8cb5
Standardize test templates for posix and s3
timj Jul 25, 2020
dba3007
Add URI quoting and unquoting
timj Jul 25, 2020
e8054f8
Use the new ButlerURI.getExtension method to work out yaml file
timj Jul 29, 2020
edec089
Make it explicit that symlink and relsymlink are being tested
timj Jul 29, 2020
da50958
Improve some docstrings in ButlerURI
timj Jul 29, 2020
e8cbcc4
Use a cast for mypy to handle the _force_to_file method
timj Jul 29, 2020
6463d05
Fail if we do not recognize the URI scheme.
timj Jul 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
21 changes: 4 additions & 17 deletions python/lsst/daf/butler/_butler.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,7 @@
ValidationError,
)
from .core.repoRelocation import BUTLER_ROOT_TAG
from .core.utils import transactional, getClassOf, safeMakeDir
from .core.s3utils import bucketExists
from .core.utils import transactional, getClassOf
from ._deferredDatasetHandle import DeferredDatasetHandle
from ._butlerConfig import ButlerConfig
from .registry import Registry, RegistryConfig, CollectionType
Expand Down Expand Up @@ -325,22 +324,10 @@ def makeRepo(root: str, config: Union[Config, str, None] = None, standalone: boo
if isinstance(config, (ButlerConfig, ConfigSubset)):
raise ValueError("makeRepo must be passed a regular Config without defaults applied.")

# for "file" schemes we are assuming POSIX semantics for paths, for
# schemeless URIs we are assuming os.path semantics.
# Ensure that the root of the repository exists or can be made
uri = ButlerURI(root, forceDirectory=True)
if uri.scheme == "file" or not uri.scheme:
if not os.path.isdir(uri.ospath):
safeMakeDir(uri.ospath)
elif uri.scheme == "s3":
# bucket must already exist
if not bucketExists(uri.netloc):
raise ValueError(f"Bucket {uri.netloc} does not exist!")
s3 = boto3.client("s3")
# don't create S3 key when root is at the top-level of an Bucket
if not uri.path == "/":
s3.put_object(Bucket=uri.netloc, Key=uri.relativeToPathRoot)
else:
raise ValueError(f"Unrecognized scheme: {uri.scheme}")
uri.mkdir()

config = Config(config)

# If we are creating a new repo from scratch with relative roots,
Expand Down
1 change: 1 addition & 0 deletions python/lsst/daf/butler/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

# Do not export the utility routines from utils and queries.

from ._butlerUri import *
from .assembler import *
from .config import *
from .configSupport import LookupKey
Expand Down