apparently publish doesn't provide --jobs to annex copy
#40
Comments
|
Ping datalad/datalad#3415 |
mih
referenced
this issue
in mih/datalad
Mar 2, 2020
- general approach is: push main branch -> copy -> push git-annex branch This will expose any history issues (missing pieces, conflicts) that could possibly invalidate local decision making. push() will fail early, allowing for fixes (e.g. update(merge=True)), and then reattempt. The annex branch is pushed last, after file transfer is completed. It is the least critical part, because annex will update availability info on the remote end on its own, as part of the transfer. - push != sync generally changes will only go from local to remote. However, in corner cases it is necessary to use `annex sync` internally to consolidate the git-annex branch or corresponding branches. - perform data transfer via async-call to `annex copy`, not via AnnexRepo.copy_to() which performs too many inspections and reporting decisions. - current approach can pass many paths to `annex copy`, so I opted for a temp file that is used as stdin for a batch-mode process of `annex copy`. This saves result merges across the alternative 'file chunk' runs. - support push to empty repos (fixes dataladgh-4074) - implement tests largely without `create_sibling`, because it doesnt work on Windows - support for managed branches - pass --jobs to git-annex copy (fixes gh-3732)
mih
referenced
this issue
in mih/datalad
Mar 3, 2020
- general approach is: push main branch -> copy -> push git-annex branch This will expose any history issues (missing pieces, conflicts) that could possibly invalidate local decision making. push() will fail early, allowing for fixes (e.g. update(merge=True)), and then reattempt. The annex branch is pushed last, after file transfer is completed. It is the least critical part, because annex will update availability info on the remote end on its own, as part of the transfer. - push != sync generally changes will only go from local to remote. However, in corner cases it is necessary to use `annex sync` internally to consolidate the git-annex branch or corresponding branches. - perform data transfer via async-call to `annex copy`, not via AnnexRepo.copy_to() which performs too many inspections and reporting decisions. - current approach can pass many paths to `annex copy`, so I opted for a temp file that is used as stdin for a batch-mode process of `annex copy`. This saves result merges across the alternative 'file chunk' runs. - support push to empty repos (fixes dataladgh-4074) - implement tests largely without `create_sibling`, because it doesnt work on Windows - support for managed branches - pass --jobs to git-annex copy (fixes gh-3732)
mih
referenced
this issue
in mih/datalad
Mar 6, 2020
- general approach is: push main branch -> copy -> push git-annex branch This will expose any history issues (missing pieces, conflicts) that could possibly invalidate local decision making. push() will fail early, allowing for fixes (e.g. update(merge=True)), and then reattempt. The annex branch is pushed last, after file transfer is completed. It is the least critical part, because annex will update availability info on the remote end on its own, as part of the transfer. - push != sync generally changes will only go from local to remote. However, in corner cases it is necessary to use `annex sync` internally to consolidate the git-annex branch or corresponding branches. - perform data transfer via async-call to `annex copy`, not via AnnexRepo.copy_to() which performs too many inspections and reporting decisions. - current approach can pass many paths to `annex copy`, so I opted for a temp file that is used as stdin for a batch-mode process of `annex copy`. This saves result merges across the alternative 'file chunk' runs. - support push to empty repos (fixes dataladgh-4074) - implement tests largely without `create_sibling`, because it doesnt work on Windows - support for managed branches - pass --jobs to git-annex copy (fixes gh-3732)
mih
referenced
this issue
in mih/datalad
Mar 8, 2020
- general approach is: push main branch -> copy -> push git-annex branch This will expose any history issues (missing pieces, conflicts) that could possibly invalidate local decision making. push() will fail early, allowing for fixes (e.g. update(merge=True)), and then reattempt. The annex branch is pushed last, after file transfer is completed. It is the least critical part, because annex will update availability info on the remote end on its own, as part of the transfer. - push != sync generally changes will only go from local to remote. However, in corner cases it is necessary to use `annex sync` internally to consolidate the git-annex branch or corresponding branches. - perform data transfer via async-call to `annex copy`, not via AnnexRepo.copy_to() which performs too many inspections and reporting decisions. - current approach can pass many paths to `annex copy`, so I opted for a temp file that is used as stdin for a batch-mode process of `annex copy`. This saves result merges across the alternative 'file chunk' runs. - support push to empty repos (fixes dataladgh-4074) - implement tests largely without `create_sibling`, because it doesnt work on Windows - support for managed branches - pass --jobs to git-annex copy (fixes gh-3732)
mih
referenced
this issue
in mih/datalad
Mar 8, 2020
- general approach is: push main branch -> copy -> push git-annex branch This will expose any history issues (missing pieces, conflicts) that could possibly invalidate local decision making. push() will fail early, allowing for fixes (e.g. update(merge=True)), and then reattempt. The annex branch is pushed last, after file transfer is completed. It is the least critical part, because annex will update availability info on the remote end on its own, as part of the transfer. - push != sync generally changes will only go from local to remote. However, in corner cases it is necessary to use `annex sync` internally to consolidate the git-annex branch or corresponding branches. - perform data transfer via async-call to `annex copy`, not via AnnexRepo.copy_to() which performs too many inspections and reporting decisions. - current approach can pass many paths to `annex copy`, so I opted for a temp file that is used as stdin for a batch-mode process of `annex copy`. This saves result merges across the alternative 'file chunk' runs. - support push to empty repos (fixes dataladgh-4074) - implement tests largely without `create_sibling`, because it doesnt work on Windows - support for managed branches - pass --jobs to git-annex copy (fixes gh-3732)
mih
referenced
this issue
in mih/datalad
Mar 9, 2020
- general approach is: push main branch -> copy -> push git-annex branch This will expose any history issues (missing pieces, conflicts) that could possibly invalidate local decision making. push() will fail early, allowing for fixes (e.g. update(merge=True)), and then reattempt. The annex branch is pushed last, after file transfer is completed. It is the least critical part, because annex will update availability info on the remote end on its own, as part of the transfer. - push != sync generally changes will only go from local to remote. However, in corner cases it is necessary to use `annex sync` internally to consolidate the git-annex branch or corresponding branches. - perform data transfer via async-call to `annex copy`, not via AnnexRepo.copy_to() which performs too many inspections and reporting decisions. - current approach can pass many paths to `annex copy`, so I opted for a temp file that is used as stdin for a batch-mode process of `annex copy`. This saves result merges across the alternative 'file chunk' runs. - support push to empty repos (fixes dataladgh-4074) - implement tests largely without `create_sibling`, because it doesnt work on Windows - support for managed branches - pass --jobs to git-annex copy (fixes gh-3732)
mih
referenced
this issue
in mih/datalad
Mar 10, 2020
- general approach is: push main branch -> copy -> push git-annex branch This will expose any history issues (missing pieces, conflicts) that could possibly invalidate local decision making. push() will fail early, allowing for fixes (e.g. update(merge=True)), and then reattempt. The annex branch is pushed last, after file transfer is completed. It is the least critical part, because annex will update availability info on the remote end on its own, as part of the transfer. - push != sync generally changes will only go from local to remote. However, in corner cases it is necessary to use `annex sync` internally to consolidate the git-annex branch or corresponding branches. - perform data transfer via async-call to `annex copy`, not via AnnexRepo.copy_to() which performs too many inspections and reporting decisions. - current approach can pass many paths to `annex copy`, so I opted for a temp file that is used as stdin for a batch-mode process of `annex copy`. This saves result merges across the alternative 'file chunk' runs. - support push to empty repos (fixes dataladgh-4074) - implement tests largely without `create_sibling`, because it doesnt work on Windows - support for managed branches - pass --jobs to git-annex copy (fixes gh-3732)
|
this issue is a twin-brother of datalad/datalad#4704 (push) which presumably (according to Closure above by @mih) was addressed in push. Since publish is still around, reopening this issue.
results in so there is no |
|
Moving this to -deprecated |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For better (faster) or for worse (discovers bugs in datalad/git-annex, might trigger site connection limits) by default we parallelize
get. But apparently we do not do that forpublish(code: https://github.com/datalad/datalad/blob/master/datalad/support/annexrepo.py#L2929). I think we should be consistent and provide similar--jobsforannex copyas wellThe text was updated successfully, but these errors were encountered: