ENH+BF: S3 and downloaders: support anonymous access + interface to add new providers #2708
Also adjusted logic on handling then auth failure so credentials get asked if anonimous access fails. Also do not add versionId option to URL if no version informatino was obtained
@@ Coverage Diff @@ ## master #2708 +/- ## ========================================= + Coverage 90.19% 90.2% +<.01% ========================================= Files 245 245 Lines 31211 31607 +396 ========================================= + Hits 28151 28510 +359 - Misses 3060 3097 +37
Refactored so UI messages make more sense
…denied Also BF access to http via basic and digest HTTP auth which is stateless, so we just need to assign .auth for the session Information for the provider is "deduced" from the failure message, although other alternatives are also given as non-default option.
* origin/master: (47 commits) DOC: Fix a docstring typo RF: show only the version, without license, upon --version DOC: Clarify datalad.__doc__'s pointer to datalad.api BF: Internal cmd call, leave error handling to outer layer DOC: Tell datalad.__doc__ readers how to see a list of commands DOC: Add summary of commands to datalad.api.__doc__ BF(TST): use .call_count since assert_called_once is N/A on older mocks RF: Move "get interface doc" code into dedicated function RF: Move "load module from interface" code to a dedicated function DOC: interface: Add a docstring to get_cmd_summaries RF: Use shorter parameter names for get_cmd_summaries RF: Move get_cmd_summaries from cmdline/main to interface/base RF: Extract CLI-independent parts from get_description_with_cmd_summary RF: cmdline: Use a dict rather than a list for short descriptions DOC: interface: Add a docstring to get_interface_groups RF: Teach get_interface_groups about plugins ENH(TST): test that ipython ui frontend gets set in under ZMQInteractiveShell ENH: dedicated test for double entry of hidden values if no choices given ENH(TST): even for hidden prompt with choices we do not reconfirm the entry ENH+BF: missing import and basic (largely smoke) test for IPython ui ...
* origin/master: (28 commits) BF: Missing import for bound method fix: added a semicolon fix: table entries get a pointer cursor to show users they can click on entry BF: Unique value hashing could not deal with lists as values BF: Recreate old GitRepo.get_git_attributes() to keep old folks going TST: Test not up to the complexity of the world DOC: Remove question that is not relevant BF: Humble attempt to fix unicode issue TST: More or less comprehensive GitRepo.(g|s)et_attributes test travis: Add a comment about grunt-contrib-qunit pinning DOC: set_gitattributes: Fix a minor typo TST: travis: Pin grunt-contrib-qunit BF: no_annex: Correct path to .gitattributes TST: Plain RF to remove 'import *' madness RF: no_annex plugin used attributes helper RF: `create` no longer hacks .gitattributes, no duplication on --force DOC: search: Fix a typo RF+ENH: New search mode egrepcs for case-sensitive search NF: GitRepo.set_gitattributes RF+ENH: Improved GitRepo.get_gitattributes ...
The default value for url is None, and re.escape with fail if called with None.
question() raises a ValueError if one of its choices isn't selected, so the program should never hit this error. Using assert here to declare this impossible is fine (and if datalad eventually works with "python -O", it's fine that the check would be dropped).
One authenticator, NoneAuthenticator, doesn't actually require authentication, so there is no need to ask for credentials.
The downstream code makes the distinction between False and None, but, with the default value of None for self.credential, there is no way for the code to evaluate to False. Assume that the intention was for needs_authentication to be set to requires_authentication (True/False) when there's an authenticator.
I can't find a way to trigger this with the tests. When a downloader class is initialized with an authenticator but no credentials, the downloader refuses to continue unless the user enters credentials. So it seems impossible to execute a codepath that assumes that there is an authenticator but no credentials.
yarikoptic merged commit
Aug 28, 2018
9 checks passed
9 checks passed
codecov/patch 90.55% of diff hit (target 90.19%)Details
codecov/project 90.2% (+<.01%) compared to 90d18e3Details
continuous-integration/appveyor/pr AppVeyor build succeededDetails
continuous-integration/travis-ci/pr The Travis CI build passedDetails
datalad-pr-dl-osx-64 DEV build done.Details
datalad-pr-docker-dl-nd14_04 DEV build done.Details
datalad-pr-docker-dl-nd16_04 DEV build done.Details
datalad-pr-docker-dl-nd80 DEV build done.Details
datalad-pr-docker-dl-nd90 DEV build done.Details
yarikoptic added a commit that referenced this pull request
Sep 13, 2018
0.10.3 (Sep 13, 2018) -- Almost-perfect This is largely a bugfix release which addressed many (but not yet all) issues of working with git-annex direct and version 6 modes, and operation on Windows in general. Among enhancements you will see the support of public S3 buckets (even with periods in their names), ability to configure new providers interactively, and improved `egrep` search backend. Although we do not require with this release, it is recommended to make sure that you are using a recent `git-annex` since it also had a variety of fixes and enhancements in the past months. Fixes - Parsing of combined short options has been broken since DataLad v0.10.0. ([#2710]) - The `datalad save` instructions shown by `datalad run` for a command with a non-zero exit were incorrectly formatted. ([#2692]) - Decompression of zip files (e.g., through `datalad add-archive-content`) failed on Python 3. ([#2702]) - Windows: - colored log output was not being processed by colorama. ([#2707]) - more codepaths now try multiple times when removing a file to deal with latency and locking issues on Windows. ([#2795]) - Internal git fetch calls have been updated to work around a GitPython `BadName` issue. ([#2712]), ([#2794]) - The progess bar for annex file transferring was unable to handle an empty file. ([#2717]) - `datalad add-readme` halted when no aggregated metadata was found rather than displaying a warning. ([#2731]) - `datalad rerun` failed if `--onto` was specified and the history contained no run commits. ([#2761]) - Processing of a command's results failed on a result record with a missing value (e.g., absent field or subfield in metadata). Now the missing value is rendered as "N/A". ([#2725]). - A couple of documentation links in the "Delineation from related solutions" were misformatted. ([#2773]) - With the latest git-annex, several known V6 failures are no longer an issue. ([#2777]) - In direct mode, commit changes would often commit annexed content as regular Git files. A new approach fixes this and resolves a good number of known failures. ([#2770]) - The reporting of command results failed if the current working directory was removed (e.g., after an unsuccessful `install`). ([#2788]) - When installing into an existing empty directory, `datalad install` removed the directory after a failed clone. ([#2788]) - `datalad run` incorrectly handled inputs and outputs for paths with spaces and other characters that require shell escaping. ([#2798]) - Globbing inputs and outputs for `datalad run` didn't work correctly if a subdataset wasn't installed. ([#2796]) - Minor (in)compatibility with git 2.19 - (no) trailing period in an error message now. ([#2815]) Enhancements and new features - Anonymous access is now supported for S3 and other downloaders. ([#2708]) - A new interface is available to ease setting up new providers. ([#2708]) - Metadata: changes to egrep mode search ([#2735]) - Queries in egrep mode are now case-sensitive when the query contains any uppercase letters and are case-insensitive otherwise. The new mode egrepcs can be used to perform a case-sensitive query with all lower-case letters. - Search can now be limited to a specific key. - Multiple queries (list of expressions) are evaluated using AND to determine whether something is a hit. - A single multi-field query (e.g., `pa*:findme`) is a hit, when any matching field matches the query. - All matching key/value combinations across all (multi-field) queries are reported in the query_matched result field. - egrep mode now shows all hits rather than limiting the results to the top 20 hits. - The documentation on how to format commands for `datalad run` has been improved. ([#2703]) - The method for determining the current working directory on Windows has been improved. ([#2707]) - `datalad --version` now simply shows the version without the license. ([#2733]) - `datalad export-archive` learned to export under an existing directory via its `--filename` option. ([#2723]) - `datalad export-to-figshare` now generates the zip archive in the root of the dataset unless `--filename` is specified. ([#2723]) - After importing `datalad.api`, `help(datalad.api)` (or `datalad.api?` in IPython) now shows a summary of the available DataLad commands. ([#2728]) - Support for using `datalad` from IPython has been improved. ([#2722]) - `datalad wtf` now returns structured data and reports the version of each extension. ([#2741]) - The internal handling of gitattributes information has been improved. A user-visible consequence is that `datalad create --force` no longer duplicates existing attributes. ([#2744]) - The "annex" metadata extractor can now be used even when no content is present. ([#2724]) - The `add_url_to_file` method (called by commands like `datalad download-url` and `datalad add-archive-content`) learned how to display a progress bar. ([#2738]) * tag '0.10.3': (214 commits) Changelog entry for 2.19 git compat fix DOC: slight tune ups to the ChangeLog ENH: link issue/pull #s in CHANGELOG, use tools/link_issues_CHANGELOG BF: remove trailing period while matching a mesage from git DOC: try_multiple_dec: Add description of `duration` parameter CLN: annexrepo: Fix grammar in a recently added comment TST: auto: Reformat comment from 900ee08 DOC: _rmtree: Drop a word from summary line DOC: Info on IDs (fixes gh-2801) BF: diff -- when reraising - just raise, do not raise that instance of exception from new location BF(TST): precommit before removing submodule so there is no batched processes ENH+BF(TST): close established ssh sockets upon test finish BF(TST): One more "clost all log handlers in the test" CHANGELOG.md: Start adding entries for v0.10.3 BF(TST): close cookies db in the teardown since atexit is later, so cannot assure no open files BF(TST): explicitly close created log handlers ENH(TST): @known_failure_windows to replace plain @skip_if_on_windows where there is hope BF(TST): do not swallow output while testing AutomagicIO to not cause some open files issue ENH(TST): Skip a test when cannot remove curent directory BF(TST): explicitly precommit a repo used under swallow_outputs ...
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments.