-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-24475: Rewrite of Butler URI and major clean up of S3 usage #336
Commits on Jul 21, 2020
-
Add more transfer modes to posix export
We really need to generalize the code that transfers one file to another location so that ingest and export use the same code and preferably can be shared amongst different datastores. You can imagine ingest from S3 to local file or export from S3 to local file or export from local file to S3. It is not scalable to put this code in multiple places in multiple datastores.
Configuration menu - View commit details
-
Copy full SHA for f02ff92 - Browse repository at this point
Copy the full SHA f02ff92View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1cd2aff - Browse repository at this point
Copy the full SHA 1cd2affView commit details -
Refactor ButlerURI to start creating subclasses
Now you get different subclasses for schemeless, file and generic URIs. This simplifies some of the logic for handling path components.
Configuration menu - View commit details
-
Copy full SHA for 54208da - Browse repository at this point
Copy the full SHA 54208daView commit details
Commits on Jul 22, 2020
-
Refactor the URI fix up code for subclasses
This now leaves us with a tiny bit of duplication between schemeless and file fixups.
Configuration menu - View commit details
-
Copy full SHA for 8287312 - Browse repository at this point
Copy the full SHA 8287312View commit details -
Defer converting to posix form until the end, thereby letting us use os.sep everywhere.
Configuration menu - View commit details
-
Copy full SHA for f926cb7 - Browse repository at this point
Copy the full SHA f926cb7View commit details
Commits on Jul 23, 2020
-
Add concrete methods for transfer and exists of URI resources
ButlerURI.transfer_from(URI) now works for file, http, and S3.
Configuration menu - View commit details
-
Copy full SHA for 0641357 - Browse repository at this point
Copy the full SHA 0641357View commit details -
Change ButlerURI.dirname so that it does not force absolute path
It is a surprise to see a schemeless relative path become a schemeless absolute path when asking for the directory component. Changing this required a couple of tests to be updated that were assuming absolute paths.
Configuration menu - View commit details
-
Copy full SHA for b2ce3f8 - Browse repository at this point
Copy the full SHA b2ce3f8View commit details -
mypy gets confused that new does not return a ButlerURI which leads to mypy thinking that we are calling abstract methods.
Configuration menu - View commit details
-
Copy full SHA for b4c8cce - Browse repository at this point
Copy the full SHA b4c8cceView commit details -
Configuration menu - View commit details
-
Copy full SHA for 12a4ddd - Browse repository at this point
Copy the full SHA 12a4dddView commit details -
Configuration menu - View commit details
-
Copy full SHA for a8f873e - Browse repository at this point
Copy the full SHA a8f873eView commit details -
Configuration menu - View commit details
-
Copy full SHA for f2c0f62 - Browse repository at this point
Copy the full SHA f2c0f62View commit details -
Configuration menu - View commit details
-
Copy full SHA for dfb206e - Browse repository at this point
Copy the full SHA dfb206eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 446f779 - Browse repository at this point
Copy the full SHA 446f779View commit details
Commits on Jul 24, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 640f65d - Browse repository at this point
Copy the full SHA 640f65dView commit details -
Configuration menu - View commit details
-
Copy full SHA for ff38951 - Browse repository at this point
Copy the full SHA ff38951View commit details -
Configuration menu - View commit details
-
Copy full SHA for ad31c1c - Browse repository at this point
Copy the full SHA ad31c1cView commit details -
Configuration menu - View commit details
-
Copy full SHA for dcbc0cd - Browse repository at this point
Copy the full SHA dcbc0cdView commit details -
Replace S3 and Resource calls with ButlerURI
Now unified resource access for files, S3, HTTP, and pkg_resources. Much simplified I/O. Still need to sort out dumpToFile
Configuration menu - View commit details
-
Copy full SHA for bbff6db - Browse repository at this point
Copy the full SHA bbff6dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for c2fe7ab - Browse repository at this point
Copy the full SHA c2fe7abView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ca5e3a - Browse repository at this point
Copy the full SHA 4ca5e3aView commit details -
For local files check if URI is a directory
This helps in cases like tempfile.mkdtemp where the path returned does not include a trailing slash but is used as a string and not a ButlerURI.
Configuration menu - View commit details
-
Copy full SHA for 86f9917 - Browse repository at this point
Copy the full SHA 86f9917View commit details -
Reimplement Config.dumpToUri to use ButlerURI.write
This removes all the special casing for dumpToFile and dumpToS3 in Config.
Configuration menu - View commit details
-
Copy full SHA for a20c8a4 - Browse repository at this point
Copy the full SHA a20c8a4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6164272 - Browse repository at this point
Copy the full SHA 6164272View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4c9227e - Browse repository at this point
Copy the full SHA 4c9227eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 38d9e32 - Browse repository at this point
Copy the full SHA 38d9e32View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8894565 - Browse repository at this point
Copy the full SHA 8894565View commit details -
Configuration menu - View commit details
-
Copy full SHA for f9bef58 - Browse repository at this point
Copy the full SHA f9bef58View commit details -
Fix ButlerURI.relative_to for schemeless vs file URIs
Absolute paths with a relative path should now work.
Configuration menu - View commit details
-
Copy full SHA for 4602a74 - Browse repository at this point
Copy the full SHA 4602a74View commit details -
Use uri relative_to method in posix datastore
At some point we should go through and require that datastore.root be a URI and not allow it to sort of be a file.
Configuration menu - View commit details
-
Copy full SHA for 94190dd - Browse repository at this point
Copy the full SHA 94190ddView commit details
Commits on Jul 25, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 53acda9 - Browse repository at this point
Copy the full SHA 53acda9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 15a4ca2 - Browse repository at this point
Copy the full SHA 15a4ca2View commit details -
Calling realpath on the destination for the transfer is never the right thing to do. It was harmless when the destination did not exist but in cases where the destination existed and was a softlink that ended up somewhere else it completely moved the output location.
Configuration menu - View commit details
-
Copy full SHA for d77ca4e - Browse repository at this point
Copy the full SHA d77ca4eView commit details -
Use local tempdir to work around relsymlink problem
on macOS the /var folder is a link to /private/var which means that when you create a temp directory in a test and put a file in it, once you readpath you end up in a completely different location that is nowhere near the test file that you are trying to use with relsymlink. Using a local tempdir fixes this anomaly.
Configuration menu - View commit details
-
Copy full SHA for 209137b - Browse repository at this point
Copy the full SHA 209137bView commit details -
Add transaction and move support to ButlerURI
This enables posixDatastore to use it.
Configuration menu - View commit details
-
Copy full SHA for c456d35 - Browse repository at this point
Copy the full SHA c456d35View commit details -
Configuration menu - View commit details
-
Copy full SHA for e0b62c3 - Browse repository at this point
Copy the full SHA e0b62c3View commit details -
Configuration menu - View commit details
-
Copy full SHA for e6a2a62 - Browse repository at this point
Copy the full SHA e6a2a62View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f5e3ca - Browse repository at this point
Copy the full SHA 0f5e3caView commit details -
Configuration menu - View commit details
-
Copy full SHA for c9181fb - Browse repository at this point
Copy the full SHA c9181fbView commit details -
Configuration menu - View commit details
-
Copy full SHA for f6b8cb5 - Browse repository at this point
Copy the full SHA f6b8cb5View commit details -
files containing ? were breaking because urllib.parse would treat this as a URI with query parameters. We therefore need to quote to protect this prior to parsing in some cases and also unquote when referring to local file resources.
Configuration menu - View commit details
-
Copy full SHA for dba3007 - Browse repository at this point
Copy the full SHA dba3007View commit details
Commits on Jul 29, 2020
-
Configuration menu - View commit details
-
Copy full SHA for e8054f8 - Browse repository at this point
Copy the full SHA e8054f8View commit details -
Make it explicit that symlink and relsymlink are being tested
Rather than looking for a substring which is easy to miss at a glance.
Configuration menu - View commit details
-
Copy full SHA for edec089 - Browse repository at this point
Copy the full SHA edec089View commit details -
Configuration menu - View commit details
-
Copy full SHA for da50958 - Browse repository at this point
Copy the full SHA da50958View commit details -
Configuration menu - View commit details
-
Copy full SHA for e8cbcc4 - Browse repository at this point
Copy the full SHA e8cbcc4View commit details -
Fail if we do not recognize the URI scheme.
This required that we also add mem:// support for in-memory datastores.
Configuration menu - View commit details
-
Copy full SHA for 6463d05 - Browse repository at this point
Copy the full SHA 6463d05View commit details