BF(TST): due to recent change in git-annex dropping sub-second clock precision - we might not report push of git-annex branch

[yarikoptic]

Comment in the test code describes it in greater detail: we ending up with identical commit locally and remotely for the content change - there is no longer difference in those subseconds we observed before:

❯ git diff git-annex^..git-annex^2
diff --git a/1d8/a44/MD5E-s12--f01b6f1223d1baf9cc6288f9172bbbf5.log b/1d8/a44/MD5E-s12--f01b6f1223d1baf9cc6288f9172bbbf5.log
index 082a191..4d55348 100644
--- a/1d8/a44/MD5E-s12--f01b6f1223d1baf9cc6288f9172bbbf5.log
+++ b/1d8/a44/MD5E-s12--f01b6f1223d1baf9cc6288f9172bbbf5.log
@@ -1,2 +1,2 @@
 1703620373.02882819s 1 00bcc85b-3ddb-49d5-b88d-41e117cf5a5b
-1703620373.778202108s 1 63dd2c48-4c33-4db3-a859-edb997189142
+1703620373.77494684s 1 63dd2c48-4c33-4db3-a859-edb997189142

and now we would have just

+++ b/1d8/a44/MD5E-s12--f01b6f1223d1baf9cc6288f9172bbbf5.log
@@ -0,0 +1 @@
+1703620331s 1 5a7c9e0d-025e-46a9-8ac1-12aa4819c28b

so no subsecond precision and commits end up identical, so whenever we fetch, we just end up on the same commit in that test.

I am marking it for release to bring our datalad/git-annex back to green state since now we have this test consistently failing across the board with such a newer git-annex.

Not sure if @joeyh foresaw/aimed exactly such possibility in change in behavior (no longer annex merge resulting in actual merge), so alerting him with this message. On the positive side -- only this test shown fragility to such a "core" change in git-annex.

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (56c2b1a) 91.56% compared to head (4bbac1c) 91.58%.

Files	Patch %	Lines
datalad/core/distributed/tests/test_push.py	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##            maint    #7544      +/-   ##
==========================================
+ Coverage   91.56%   91.58%   +0.02%     
==========================================
  Files         325      325              
  Lines       43445    43454       +9     
  Branches     5827     5785      -42     
==========================================
+ Hits        39781    39798      +17     
+ Misses       3649     3641       -8     
  Partials       15       15              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[joeyh]

Yaroslav Halchenko wrote:

Not sure if @joeyh foresaw/aimed exactly such possibility in change in behavior (no longer annex merge resulting in actual merge), so alerting him with this message. On the positive side -- only this test shown fragility to such a "core" change in git-annex.

What I anticipated is that, potentially, a situation where conflicting changes are made at very close to the same time, to a config value such as `git-annex numcopies`, there can be a behavior change in which of the two conflicting values wins out. Since a small amount of clock skew would have a similar result, and the same person doesn't typically make conflicting config changes at the same time, I didn't feel this was enough of a problem to avoid the optimisation. In your case, there's no conflicting change being made (indeed changes to the location log are non-conflicting almost always). But I think this behavior of the same commit being made is fine and it makes sense you adjust your tests. You could use GIT_ANNEX_VECTOR_CLOCK in your test case with two different values to force two different commits to be made.

…

-- see shy jo

[yarikoptic]

You could use GIT_ANNEX_VECTOR_CLOCK in your test case with two different values to force two different commits to be made.

I believe not in this particular case -- it is a matter of both commits to be done by git-annex copy - one locally and one on "remote"... here is my reproduction attempt -- I might be using that env var incorrectly through but it shows (no difference) between local and remote commits with current version and with prior one smth like

difference between local and remote git-annex commits:
--- /dev/fd/63	2023-12-28 11:48:40.094850111 -0500
+++ /dev/fd/62	2023-12-28 11:48:40.094850111 -0500
@@ -1,13 +1,13 @@
-commit 31d4112d2bd0aff16c101f7fbcc52af6a9995fa9
+commit 9fb7a8ede8eb33c6374fd0980422b655212262ae
 Author: Yaroslav Halchenko <debian@onerussian.com>
 Date:   Thu Dec 28 11:48:40 2023 -0500
 
     update
 
 diff --git a/46c/72d/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b.log b/46c/72d/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b.log
-index 21ba826..bfdb0f9 100644
+index 21ba826..5e644a9 100644
 --- a/46c/72d/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b.log
 +++ b/46c/72d/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b.log
 @@ -1 +1,2 @@
  1703782119.810539895s 1 37202dd9-c8bc-48ae-9f57-f3efd914ebac
-+1703782120.055025347s 1 8c76fcb4-b519-491b-863a-11e0d3a58aa0
++1703782120.049070504s 1 8c76fcb4-b519-491b-863a-11e0d3a58aa0

reproduction script

#!/bin/bash

cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)"
set -eu

(
mkdir push-target
cd push-target
git init --bare
git annex init
)

(
mkdir origin
cd origin
git init
git annex init

echo 123 > 123
git annex add 123
git commit -m 'commit 123'

git remote add --fetch push-target ../push-target
git annex merge
git push push-target git-annex
GIT_ANNEX_VECTOR_CLOCK=2170378167 git annex copy --to=push-target --all

echo -e "\ndifference between local and remote git-annex commits:"
diff -Naur <(git show git-annex) <(git -C ../push-target show git-annex)
)

[yarikoptic]

meanwhile I will merge/release to bring our daily tests into green... interestingly that on windows on released version of datalad failed the test_force_checkdatapresent test -- the other two (maint and master) seems to be green on the last run, and on previous master was green -- odd... will remain mystery ;-)

[yarikoptic-gitmate]

PR released in 0.19.5