Retire distribution.create and promote rev_create #3383

kyleam · 2019-05-03T16:26:20Z

Remaining things to do:

adjust benchmarks

Hmm, based off of

Lines 11 to 17 in df1596c

    
           try: 
        
               from datalad.api import rev_save 
        
               from datalad.api import rev_create 
        
           except ImportError: 
        
               # If it is a version without revolution - those will not be benchmarked 
        
               pass

perhaps there's not anything that requires adjustment.

adjust -revolution to provide rev_create alias
adjust -crawler (https://travis-ci.org/datalad/datalad/jobs/527865603#L1891)

Closes #3379.

There is no additional commit in the submodule; it just has staged changes.

Unlike distribution.create, rev_create puts a .noannex file in the repository if --no-annex was given. Adjust test_unlock_raises to remove the .noannex file before trying to convert the existing repository into an annex repository.

8389502 (RF: internal use of `create` -> `rev_create`, 2019-03-22) converted most spots, but there are still a few places outside of distribution/tests/test_create.py where we use distribution.create. Convert the remaining spots, leaving only a few in test_add.py (to be addressed next) and the ones in the benchmarks.

These tests rely on create's save argument, which rev_create dropped. In test_add_dirty_tree, the save=False is unnecessary, so use a plain rev_create call. In the other spots, use an unbound rev_create call to avoid saving the subdataset in the superdataset. The results aren't identical, but the differences don't matter in the context of these tests.

Closes datalad#3379.

[skip travis]

codecov · 2019-05-03T16:57:28Z

Codecov Report

Merging #3383 into master will decrease coverage by 0.15%.
The diff coverage is 99.44%.

@@            Coverage Diff             @@
##           master    #3383      +/-   ##
==========================================
- Coverage   91.24%   91.08%   -0.16%     
==========================================
  Files         265      263       -2     
  Lines       34452    34152     -300     
==========================================
- Hits        31436    31109     -327     
- Misses       3016     3043      +27

Impacted Files	Coverage Δ
datalad/interface/__init__.py	`100% <ø> (ø)`	⬆️
datalad/customremotes/tests/test_archives.py	`89.47% <0%> (-0.6%)`	⬇️
datalad/interface/tests/test_utils.py	`100% <100%> (ø)`	⬆️
datalad/distribution/tests/test_add.py	`82.95% <100%> (+0.46%)`	⬆️
datalad/metadata/extractors/tests/test_exif.py	`93.54% <100%> (ø)`	⬆️
datalad/plugin/tests/test_plugins.py	`89.02% <100%> (ø)`	⬆️
datalad/support/tests/test_repo_save.py	`100% <100%> (ø)`	⬆️
datalad/distribution/tests/test_create_github.py	`93.02% <100%> (ø)`	⬆️
datalad/distribution/tests/test_publish.py	`100% <100%> (ø)`	⬆️
datalad/interface/tests/test_run_procedure.py	`100% <100%> (ø)`	⬆️
... and 43 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca35877...9cced04. Read the comment docs.

With rev-create (now create), anything after the path is taken as an option to 'git init'.

datalad/datalad#3383 replaces core's create with rev-create. Add back a compatibility alias, like we already do for diff and status.

mih · 2019-05-03T19:19:20Z

The diff looks pleasantly straightforward. Thx.

kyleam · 2019-05-04T02:54:55Z

The crawler used the backend argument of the old create to set the backend to datalad.crawl.default_backend. The new create relies soley on config values, which works nicely from the command line:

datalad -c datalad.repo.backend=SHA256E create blah

What's the cleanest way for a python caller to set this for one invocation of create? The two best options I can think of, neither pretty IMO, are (1) temporarily overriding the environment variable and reloading datalad.cfg or (2) using a fresh config instance instead of datalad.cfg and specifying overrides. Anything better that I'm overlooking?

mih · 2019-05-04T06:20:48Z

What's the cleanest way for a python caller to set this for one invocation of create?

I would take one further step back and ignore the create default entirely. I see no problem or disadvantage of resetting the value after create is done, as this is a setting that lives in the local clone, and does not affect any content in the create commit.

Hence:

>>> import datalad.api as dl
>>> ds = dl.rev_create('.')
[INFO   ] Creating a new annex repo at /tmp/backend

% cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[annex]
        uuid = a878132b-1be5-41e0-923d-74c1754b02ef
        version = 5
        backends = MD5E

# replace value SHA256E with value of `datalad.crawl.default_backend`
>>> ds.config.set('annex.backends', 'SHA256E', where='local')

% cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[annex]
        uuid = a878132b-1be5-41e0-923d-74c1754b02ef
        version = 5
        backends = SHA256E

So that is doable and less complicated than any global overrides. However, I wonder why the crawler needs a different default, and what requirements it has that demand it to be different than MD5E -- and whether we should have the same default in -core?

Looking into this I see:

datalad_crawler/nodes/annex.py:        backend = self.backend or cfg.obtain('datalad.crawl.default_backend', default='MD5E')

So it seems to me that those settings are redundant, the requirements are identical, and datalad.crawl.default_backend can be retired in favor of the more general datalad.repo.backend.

@yarikoptic what do you think?

kyleam · 2019-05-04T14:35:57Z

I see no problem or disadvantage of resetting the value after create is done, as this is a setting that lives in the local clone, and does not affect any content in the create commit.

It is more than just a setting in the local clone. Both the old create and rev-create set the annex backend in .gitattributes with set_default_backend(..., persistent=True). With your suggestion to change git/.config, adding new files would stil honor the value set in .gitattributes:

import os
import tempfile
import datalad.api as dl

ds = dl.create(tempfile.mkdtemp(prefix="dl-"))  # rev_create
ds.config.set('annex.backends', 'SHA256E', where='local')
(ds.pathobj / "blah").write_text("ooo")
ds.rev_save()
print(os.readlink(str(ds.pathobj / "blah")))
# .git/annex/objects/37/Mv/MD5E-s3--7f94dd413148ff9ac9e9e4b6ff2b6ca9/MD5E-s3--7f94dd413148ff9ac9e9e4b6ff2b6ca9

Before asking about this, I briefly explored doing it after, with something like

diff --git a/datalad_crawler/nodes/annex.py b/datalad_crawler/nodes/annex.py
index 442397f..af802d3 100644
--- a/datalad_crawler/nodes/annex.py
+++ b/datalad_crawler/nodes/annex.py
@@ -138,8 +138,6 @@ def _initiate_dataset(self, path, name):
             # TODO: RF whenevever create becomes a dedicated factory/method
             # and/or branch becomes an option for the "creator"

-        backend = self.backend or cfg.obtain('datalad.crawl.default_backend', default='MD5E')
-
         ds = create(
                 path=path,
                 force=False,
@@ -148,11 +146,14 @@ def _initiate_dataset(self, path, name):
                 #  custom backend was specified, but now with dataset id -- should always save
                 # save=not bool(backend),
                 # annex_version=None,
-                annex_backend=backend,
                 #git_opts=None,
                 #annex_opts=None,
                 #annex_init_opts=None
         )
+        ds.repo.set_default_backend(
+            self.backend or cfg.obtain(
+                'datalad.crawl.default_backend', default='MD5E'),
+            persistent=True, commit=True)
         if self.add_to_super:
             # place hack from 'add-to-super' times here
             # MIH: tests indicate that this wants to discover any dataset above

But set_default_backend() won't override an existing value (perhaps it should learn a force option or just do so if there are no annex objects yet?).

mih · 2019-05-05T16:15:16Z

Sorry, I missed the attributes. I see no better way than using cfg overrides (as you initially pointed out). You can also dump a file with just * annex.backend=SHA256 in the directory, and run create with --force. But I would not call this pretty either.

But @yarikoptic's feedback is still out on why having this second same default to begin with.

yarikoptic · 2019-05-05T21:14:55Z

I think that ideally all python API commands should support passing dictionary of configuration variables, to be in line with command line API.

As for crawler, we could indeed retire it anyone asks for it back. I typically really to default anyways for any new dataset

rev-create, soon to be the main create, doesn't allow configuring the backend via an argument. To be compatible with the new create, we could set datalad.repo.backend to whatever value is configured via a crawler-specific mechanism. But it's not clear that any crawler-specific mechanisms are needed to configure the backend, so instead warn that the crawler setting isn't honored and point to datalad.repo.backend. Re: datalad/datalad#3383 (comment)

kyleam · 2019-05-06T16:13:38Z

This PR and the sister PRs (datalad/datalad-revolution#121 and datalad/datalad-crawler#41) are ready to be reviewed.

mih

Looks great! I was hoping for the diff to be like this, and now I am happy that it actually worked out like this. Thanks much for taking care of this PR.

mih · 2019-05-06T18:15:42Z

datalad/datalad-revolution#121 is now waiting for this to be merged. I think an rc3 release is also due after the merge.

mih · 2019-05-07T05:12:38Z

@yarikoptic I will merge this now in order to get rc3 out as a depencency for the metadata work. Given the nature of the diff (almost exclusively a renaming) I think the chances for unexpected things a low. And we can always adjust later on.

kyleam added 6 commits May 3, 2019 11:42

TST: add: Correct a comment

3dafce1

There is no additional commit in the submodule; it just has staged changes.

TST: unlock: Use rev_create

35b0d0c

Unlike distribution.create, rev_create puts a .noannex file in the repository if --no-annex was given. Adjust test_unlock_raises to remove the .noannex file before trying to convert the existing repository into an annex repository.

RF: Retire distribution.create and promote rev_create

6f8c54f

Closes datalad#3379.

TST: appveyor.yml: Update for distribution.create's removal

97c5079

[skip travis]

DOC: 3rdparty_analysis_workflow: Update create call

8144048

With rev-create (now create), anything after the path is taken as an option to 'git init'.

kyleam added the WIP work in progress label May 3, 2019

TST: travis: Temporarily disable -crawler run

9cced04

kyleam added a commit to kyleam/datalad-revolution that referenced this pull request May 3, 2019

Resurrect rev-create as shim for core's create

73eb2c7

datalad/datalad#3383 replaces core's create with rev-create. Add back a compatibility alias, like we already do for diff and status.

kyleam mentioned this pull request May 3, 2019

Update for rev-create rename in core datalad/datalad-revolution#121

Merged

kyleam mentioned this pull request May 6, 2019

Update crawler for rev-create promotion datalad/datalad-crawler#41

Merged

kyleam removed the WIP work in progress label May 6, 2019

mih approved these changes May 6, 2019

View reviewed changes

kyleam mentioned this pull request May 6, 2019

Add CHANGELOG entries for 0.12rc3 #3387

Merged

mih merged commit b8ca6c9 into datalad:master May 7, 2019

This was referenced May 7, 2019

"backend" commit slows create operations #2911

Closed

Adding new (deep) subdatasets broken on win v6 #2857

Closed

--git-opts are not just ignored -- they interfere for create --shared-access #2818

Closed

This was referenced May 7, 2019

Pass --annex-init-opts to git annex init - do not ignore them #2851

Closed

Put a .noannex file in repo with create --no-annex #2669

Closed

unable to create new dataset with 'rev-create' #3305

Closed

kyleam deleted the replace-create branch May 7, 2019 13:11

yarikoptic added this to the Release 0.12.0 milestone May 7, 2019

kyleam mentioned this pull request Jul 1, 2019

TST: travis: Re-enable run for crawler extension #3504

Closed

kyleam mentioned this pull request Oct 2, 2020

have an easy way to set/change default backend for annex #4978

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retire distribution.create and promote rev_create #3383

Retire distribution.create and promote rev_create #3383

kyleam commented May 3, 2019 •

edited

codecov bot commented May 3, 2019 •

edited

mih commented May 3, 2019

kyleam commented May 4, 2019

mih commented May 4, 2019

kyleam commented May 4, 2019

mih commented May 5, 2019

yarikoptic commented May 5, 2019

kyleam commented May 6, 2019

mih left a comment

mih commented May 6, 2019

mih commented May 7, 2019


	try:
	from datalad.api import rev_save
	from datalad.api import rev_create
	except ImportError:
	# If it is a version without revolution - those will not be benchmarked
	pass

Retire distribution.create and promote rev_create #3383

Retire distribution.create and promote rev_create #3383

Conversation

kyleam commented May 3, 2019 • edited

codecov bot commented May 3, 2019 • edited

Codecov Report

mih commented May 3, 2019

kyleam commented May 4, 2019

mih commented May 4, 2019

kyleam commented May 4, 2019

mih commented May 5, 2019

yarikoptic commented May 5, 2019

kyleam commented May 6, 2019

mih left a comment

Choose a reason for hiding this comment

mih commented May 6, 2019

mih commented May 7, 2019

kyleam commented May 3, 2019 •

edited

codecov bot commented May 3, 2019 •

edited