Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RF: Blind attempt to relay status() performance boost possibility on to save() #3712

Merged
merged 2 commits into from Sep 30, 2019

Conversation

mih
Copy link
Member

@mih mih commented Sep 25, 2019

According to my current logic and assessment, the status() run done in
top-level save() does not need to inspect filetypes. The tests will show
if that is true.

If true, performance gains of the same magnitude as shown in
#3701 for status() also become
available for save().

@mih mih added the performance label Sep 25, 2019
@mih mih changed the title RF: Blind attempt to relay status() performance possibility on to save() RF: Blind attempt to relay status() performance boost possibility on to save() Sep 25, 2019
mih added 2 commits Sep 26, 2019
According to my current logic and assessment, the status() run done in
top-level save() does not need to inspect filetypes. The tests will show
if that is true.

If true, performance gains of the same magnitude as shown in
datalad#3701 for status() also become
available for save().
python -m nose -s -v datalad.support.tests.test_annexrepo:test_files_split

Before:
249.66s user 21.72s system 139% cpu 3:14.46 total

After:
199.29s user 20.01s system 129% cpu 2:48.96 total
@mih
Copy link
Member Author

@mih mih commented Sep 26, 2019

Hmm, strange timeouts on travis. I cannot reproduce that locally and travis no longer tells which particular test it is happening in.

@codecov
Copy link

@codecov codecov bot commented Sep 29, 2019

Codecov Report

Merging #3712 into master will decrease coverage by 2.27%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #3712      +/-   ##
=========================================
- Coverage   82.77%   80.5%   -2.28%     
=========================================
  Files         273     273              
  Lines       35918   35918              
=========================================
- Hits        29732   28915     -817     
- Misses       6186    7003     +817
Impacted Files Coverage Δ
datalad/support/tests/test_annexrepo.py 95.7% <ø> (-0.24%) ⬇️
datalad/core/local/save.py 87.83% <ø> (ø) ⬆️
datalad/metadata/aggregate.py 14.24% <0%> (-43.92%) ⬇️
datalad/metadata/metadata.py 45.15% <0%> (-42.94%) ⬇️
datalad/metadata/search.py 35.39% <0%> (-41.34%) ⬇️
datalad/interface/save.py 25.49% <0%> (-39.87%) ⬇️
datalad/distribution/add.py 27.8% <0%> (-34.23%) ⬇️
datalad/support/archives.py 61.32% <0%> (-22.66%) ⬇️
datalad/interface/run_procedure.py 67.08% <0%> (-21.74%) ⬇️
datalad/api.py 75.86% <0%> (-17.25%) ⬇️
... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 040eee2...b8d5823. Read the comment docs.

@mih
Copy link
Member Author

@mih mih commented Sep 30, 2019

OK, whatever it would break, our tests don't see it. Neither does my imagination. So let's do it and wait for the eventual bug report.

@mih mih merged commit b950066 into datalad:master Sep 30, 2019
4 of 5 checks passed
@mih mih deleted the rf-saveperf branch Sep 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant