git tree : duplicate entries on push

[SvSky]

Hello,

i try to push the master branch to another repo but i can't and have this error.

git push <repo_name> master (no master branch on this repo, but master.old (2.5beta30)):
Décompte des objets: 91706, fait.
Delta compression using up to 8 threads.
Compression des objets: 100% (19007/19007), fait.
remote: error: object ffa998510f3e1206ac95a4a3e35e25e3f4017adb: duplicateEntries: contains duplicate file entries
remote: fatal: fsck error in packed object
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error
Écriture des objets: 100% (91706/91706), 53.33 MiB | 14.55 MiB/s, fait.
Total 91706 (delta 61011), reused 91690 (delta 61008)
error: le dépaquetage a échoué : index-pack abnormal exit
Everything up-to-date

git fsck --full :
Vérification des répertoires d'objet: 100% (256/256), fait.
error in tree b7a0b493966f9077f6fcd7de37244699bb73615e: duplicateEntries: contains duplicate file entries
error in tree 2f5da62ad954c1d90c4618f1277d0f5474bed327: duplicateEntries: contains duplicate file entries
error in tree ffa998510f3e1206ac95a4a3e35e25e3f4017adb: duplicateEntries: contains duplicate file entries
Vérification des objets: 100% (92209/92209), fait.

git ls-tree --full-tree -rt -l > .txt :
2f5da62ad954c1d90c4618f1277d0f5474bed327.txt
b7a0b493966f9077f6fcd7de37244699bb73615e.txt
ffa998510f3e1206ac95a4a3e35e25e3f4017adb.txt

Thank you

[J-Jamet]

Issue comes from old repo : bpellin/keepassdroid#441
I guess you already looked at this thread:
https://stackoverflow.com/questions/13178720/tree-contains-duplicate-file-entries
https://stackoverflow.com/questions/10931954/git-tree-contains-duplicate-file-entries

To remove dangling blob & commit : git reflog expire --expire=now --all && git gc --prune=now --aggressive

I think Brian did some tests on Windows and committed files with the same names but different case. I tried several commands to fix these trees but I would have to rewrite the whole history to make it work. So it will stay like this. What are you trying to push and on which repo, I don't see any public project on your github.

[SvSky]

Hello,
I try to push on a gitlab repo (work environment)

It's an old issue but i solve it :
mkdir KeePassDXtmp
cd KeePassDXtmp
git init
cd ../KeePassDX
git fast-export --all | (cd ../KeePassDXtmp && git fast-import)
cd ../KeePassDXtmp
git remote add (my repo)
git push

and it worked

[licaon-kter]

Seeing the same issue on clone too: https://monitor.f-droid.org/builds/log/com.kunzisoft.keepass.libre/125#site-footer

F-Droid can't build 4.0.2, fyi https://gitlab.com/fdroid/fdroiddata/-/commit/8c7b10af747f08820e243f4b3880015fa1fd1dd9

/LE: ref: https://gitlab.com/fdroid/fdroidserver/-/issues/1084

[J-Jamet]

Hello, I try to push on a gitlab repo (work environment)

It's an old issue but i solve it : mkdir KeePassDXtmp cd KeePassDXtmp git init cd ../KeePassDX git fast-export --all | (cd ../KeePassDXtmp && git fast-import) cd ../KeePassDXtmp git remote add (my repo) git push

and it worked

I can't afford to rewrite the history and change the hashes of all the commits.
I need to find a solution that only solves the problem at the level of the duplicate tree.

I used to have no trouble building on F-Droid with this repo, so something must have changed.

[cjc7373]

AFAIK the only way to fix this is to rewrite the whole history. Github/F-droid server has some git settings like transfer.fsckObjects set, so they run git fsck when reciveing/fetching objects, that's why the error happens.

I think f-droid people may tweak some settings to make the clone success. The link posted above already pointed out some ways to ignore this kind of error.

[licaon-kter]

Actually it does not, as the setting is VM wise, the first clone fails.

For other apps the issue was with some submodule-d repo that was cloned later hence we could have controlled that manually.

For now there's no solution, F-Droid wise...

[shuvashish76]

Since the app is not set up for reproducible builds at F-Droid, we can't even use izzyrepo for updates. So basically F-Droid users stuck with v3.5.1.

Edit: OK the package IDs are also different.

[IzzySoft]

I'll pick up the version offered here meanwhile as it doesn't conflict (thanks @shuvashish76 for the detailed report!) I'll also pick up Fastlane with future releases, but for now have to skip full_description.txt due to some formatting issue: my repo renders it via standard (not Github-flavored) Markdown, so it needs an empty line also before the bullet-point list (and ideally also the space in front of each bullet point removed). If you can adjust that and let me know afterwards @J-Jamet, I'd pick it up as well.

[J-Jamet]

@IzzySoft Can you make a pull request to fix the full description?

I've done a lot of testing and I've identified which commits are at fault, and they're really at the very beginning of the project. :'(

Faulty commits :

aae92abb0e51299c77610b1187103117d4a6142e
a85773c9cfc0f23ced5b0069304c799573e8cc46
d893a82970e6010f10079f2d81192960b637ce76

Found with command :
git whatchanged --all --find-object=7f457b2ab1e120a05801927bd6cd3936c330ea69

7f457b2ab1e120a05801927bd6cd3936c330ea69 is a blob of duplicated file.

I'd like to be able to squash these commits to delete them, but that would rewrite the history. Brian was testing and wrote some duplicated files in git but they are not necessary so it doesn't matter if these commits are removed.

Maybe with a git replace by deleting the old history and pointing it to the new one, but I don't know if that will solve the problem and keep the hashes.

If anyone has a bright idea that would be great.

[Uj947nXmRqV2nRaWshKtHzTvckUUpD]

https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository
also: https://stackoverflow.com/questions/30893040/remove-commit-from-history

could these help?
! careful though - from my understanding, it requires coordination with collaborators

[GalaxySnail]

Maybe with a git replace by deleting the old history and pointing it to the new one, but I don't know if that will solve the problem and keep the hashes.

Unfortunately AFAIK it's not possible to fix it without rewriting the history. According to the documentation, git replace has no effect on git fsck.

Is it currently possible to configure fsck.skipList or fsck.duplicateEntries ignore on f-droid?

[eighthave]

https://gitlab.com/fdroid/fdroid-bootstrap-buildserver/-/merge_requests/30 will give the fdroiddata maintainers the power to add per-commit exceptions to the fsck checks. It seems like there will be three commit IDs the KeePassDX maintainers should add to the new files, e.g. https://gitlab.com/fdroid/fdroiddata/-/merge_requests/14014