Core Repo/Dataset classes #4689

mih · 2020-07-11T11:43:25Z

Merely the first steps towards a resolution of gh-3988 in the spirit of gh-3192, hopefully and eventually leading to a sliming/modernization of the public API (gh-4589).

There is not much happening here, other than moving code around and establishing the classes. The plan is to lower the threshold to transition some carefully filtered functionality in the core API. At the same time the main Repo classes will be used primarily for a long time still. But it should become progressively easier to write code that only uses the core API.

Major pieces:

core repo classes GitRepo and AnnexRepo (i.e. identically named as suggested in Establish core *Repo classes #3988 (comment))
call_git*() methods have moved and no longer rely on an execution helper that invokes normalize_paths()
transition a minimal additional set of functionality to make the flyweight pattern work with the core API only
not moving code that directly uses wrapt

Fixes gh-3988

I am currently pondering whether we might also want to established corresponding locations for core utils, exceptions and support with the analog goal of a rejuvenation/slimming of the code base. But later and in a separate effort.

mih · 2020-07-12T07:40:52Z

This is now about as far as I'd go for a meaningful first step.

bpoldrack

Looks good to me as a first move, except for the os.path issue I commented on. I think at least we shouldn't encourage users of the python API (or ourselves for that matter) to use it over pathobj.

datalad/core/dataset/dataset.py

datalad/core/dataset/gitrepo.py

codecov · 2020-07-13T19:43:10Z

Codecov Report

Merging #4689 into master will decrease coverage by 0.12%.
The diff coverage is 95.77%.

@@            Coverage Diff             @@
##           master    #4689      +/-   ##
==========================================
- Coverage   89.71%   89.58%   -0.13%     
==========================================
  Files         289      291       +2     
  Lines       40455    40231     -224     
==========================================
- Hits        36295    36042     -253     
- Misses       4160     4189      +29

Impacted Files	Coverage Δ
datalad/support/gitrepo.py	`89.55% <89.65%> (-0.95%)`	⬇️
datalad/core/dataset/utils.py	`90.47% <90.47%> (ø)`
datalad/core/dataset/annexrepo.py	`90.90% <90.90%> (ø)`
datalad/core/dataset/dataset.py	`96.84% <96.84%> (ø)`
datalad/core/dataset/gitrepo.py	`97.14% <97.14%> (ø)`
datalad/core/dataset/__init__.py	`96.77% <100.00%> (ø)`
datalad/core/local/repo.py	`100.00% <100.00%> (+9.52%)`	⬆️
datalad/customremotes/tests/test_archives.py	`89.93% <100.00%> (-0.07%)`	⬇️
datalad/distribution/dataset.py	`93.29% <100.00%> (-1.13%)`	⬇️
...lad/distribution/tests/test_create_test_dataset.py	`100.00% <100.00%> (ø)`
... and 69 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d4d788...7c61a0a. Read the comment docs.

kyleam

Glancing through --color-moved diffs, the code movement looks fine to me, though admittedly my eyes glazed over at some point.

datalad/core/dataset/gitrepo.py

datalad/support/gitrepo.py

mih · 2020-07-14T09:56:11Z

I pushed a series of commits that address all raised points. I will evaluate these changes in the context of #4698 now.

bpoldrack · 2020-07-15T07:23:37Z

Seems you stripped path from GitRepo but not from Dataset. Oversight or is there a reason for it?

This is following up on the plan made in dataladgh-3192, and elaborated on in dataladgh-3988 and dataladgh-4589 (establish carefully filtered, minimized core repo classes).

Better fit with new structure

With the absolute bare essentials and pretty much no real functionality.

And a whole slew of changes that was necessary to be able to run AnnexRepo.is_valid_annex() in this new class.

and RF them to only rely on core functionality.

mih · 2020-08-27T13:31:35Z

Rebased on master after an eternity has passed.

This is recovering the first set of changes from the stale dataladgh-4689

Follow-up to dataladgh-4841 to eventually fulfill dataladgh-4689 Diff looks complicated but it is basically moving the function body into the `try` block to be able to use `call_git_items_()`

But use `call_git*()` methods instead. Follow-up to dataladgh-4841 to eventually fulfill dataladgh-4689

Follow-up to dataladgh-4841 to eventually fulfill dataladgh-4689

All `call_git*()` methods are now running `GitWitlessRunner`. Moreover, there is no `normalize_paths` decorator involvement for any such invokations anymore (ping dataladgh-3881, dataladgh-4595). For now `GitRepo._git_custom_command()` is kept -- at least until dataladgh-4847 and dataladgh-4848 are merged. Follow-up to dataladgh-4841 to eventually fulfill dataladgh-4689

mih · 2020-10-04T11:11:45Z

This will need to be attempted at a later point again. Too much has changed.

mih changed the title ~~Core Repo classes~~ Core Repo/Dataset classes Jul 11, 2020

mih force-pushed the rf-corerepo branch from e7b925a to a43c782 Compare July 12, 2020 07:33

This was referenced Jul 12, 2020

Restructure source tree #3192

Closed

Large (number of files) save operations are extremely slow and resource intensive #4630

Closed

bpoldrack reviewed Jul 13, 2020

View reviewed changes

datalad/core/dataset/dataset.py Show resolved Hide resolved

datalad/core/dataset/gitrepo.py Outdated Show resolved Hide resolved

mih force-pushed the rf-corerepo branch from a43c782 to 8094be3 Compare July 13, 2020 18:42

mih mentioned this pull request Jul 13, 2020

GitWitless-based call_git() and wipe out _git_custom_command() #4698

Closed

kyleam reviewed Jul 13, 2020

View reviewed changes

mih force-pushed the rf-corerepo branch from 8094be3 to 359b3bf Compare July 14, 2020 09:54

mih added 17 commits August 27, 2020 15:18

RF: datalad/support/repo.py -> datalad/core/dataset/__init__.py

5a99736

This is following up on the plan made in dataladgh-3192, and elaborated on in dataladgh-3988 and dataladgh-4589 (establish carefully filtered, minimized core repo classes).

RF: datalad/core/local/repo.py -> datalad/core/dataset/utils.py

86c062e

Better fit with new structure

RF: Add compatibility module to ensure seamless upgrades

d362d5b

RF: Normalize imports

3ff4b69

RF: Establish core GitRepo class

2f7bc20

With the absolute bare essentials and pretty much no real functionality.

RF: Cleanup imports

db12cd9

RF: core AnnexRepo class

05d3ef8

And a whole slew of changes that was necessary to be able to run AnnexRepo.is_valid_annex() in this new class.

RF: Split off core Dataset

34a8887

RF: Move call_git*() methods to core GitRepo

c08833c

and RF them to only rely on core functionality.

RF: Replace trivial invocations of _git_custom_command() with call_git()

e4aff83

Move fake dates property

b38eedd

Remove time import

e6b9dfb

Remove honor_fake dates

963cdad

Remove index file parameter

3ec4f4d

Remove env marker for fake dates

5733e05

More robust handling of recursion prevention

13e5f1c

Strip .path from core GitRepo (only .pathobj remains)

7c61a0a

mih force-pushed the rf-corerepo branch from 359b3bf to 7c61a0a Compare August 27, 2020 13:30

mih added a commit to mih/datalad that referenced this pull request Aug 28, 2020

RF: Replace trivial usage of _git_custom_command with call_git*()

073d907

This is recovering the first set of changes from the stale dataladgh-4689

mih mentioned this pull request Aug 28, 2020

RF: Replace trivial usage of _git_custom_command with call_git*() #4841

Merged

mih mentioned this pull request Aug 31, 2020

RF: Remove last usage of GitRepo._git_custom_command from AnnexRepo #4847

Merged

mih added a commit to mih/datalad that referenced this pull request Aug 31, 2020

RF: Test helpers no longer use GitRepo._git_custom_command

7165a4e

But use `call_git*()` methods instead. Follow-up to dataladgh-4841 to eventually fulfill dataladgh-4689

mih mentioned this pull request Aug 31, 2020

RF: Test helpers no longer use GitRepo._git_custom_command #4848

Merged

mih added a commit to mih/datalad that referenced this pull request Aug 31, 2020

RF: No _git_custom_command usage in gitrepo.py outside call_git helpers

0dcdbfa

Follow-up to dataladgh-4841 to eventually fulfill dataladgh-4689

mih mentioned this pull request Aug 31, 2020

Remove GitRepo._git_custom_command usage from call_git() methods #4850

Merged

mih added the stale-PR-closed-without-merge label Oct 4, 2020

mih closed this Oct 4, 2020

mih deleted the rf-corerepo branch September 24, 2021 10:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core Repo/Dataset classes #4689

Core Repo/Dataset classes #4689

mih commented Jul 11, 2020 •

edited

Loading

mih commented Jul 12, 2020

bpoldrack left a comment

codecov bot commented Jul 13, 2020 •

edited

Loading

kyleam left a comment

mih commented Jul 14, 2020

bpoldrack commented Jul 15, 2020

mih commented Aug 27, 2020

mih commented Oct 4, 2020

Core Repo/Dataset classes #4689

Core Repo/Dataset classes #4689

Conversation

mih commented Jul 11, 2020 • edited Loading

mih commented Jul 12, 2020

bpoldrack left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 13, 2020 • edited Loading

Codecov Report

kyleam left a comment

Choose a reason for hiding this comment

mih commented Jul 14, 2020

bpoldrack commented Jul 15, 2020

mih commented Aug 27, 2020

mih commented Oct 4, 2020

mih commented Jul 11, 2020 •

edited

Loading

codecov bot commented Jul 13, 2020 •

edited

Loading