Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When cloning a repository: "BUG: refs.c:1119: create called without valid new_oid" #3781

Closed
helarsen opened this issue Apr 10, 2022 · 6 comments

Comments

@helarsen
Copy link

  • [x ] I was not able to find an open or closed issue matching what I'm seeing

Setup

Git 32bit, but also Git 64bit gives same result

$ git --version --build-options
git version 2.35.1.windows.2
cpu: i686
built from commit: 5437f0fd368c7faf1a0b5e1fef048232c1f2a3e6
sizeof-long: 4
sizeof-size_t: 4
shell-path: /bin/sh
feature: fsmonitor--daemon

$ cmd.exe /c ver
Microsoft Windows [Version 10.0.19042.1620]

$ type "C:\Program Files (x86)\Git\etc\install-options.txt"
Editor Option: Notepad++
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Merge
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled
Enable FSMonitor: Disabled
  • Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other
Bash and tortoiseGit - same result
  • When trying to clone a private and file-hosted repository like this:
$ git clone -v --progress "G:\\My Drive\\Git Repos\\o_p_aux.git" o_p_aux
Cloning into 'o_p_aux'...
done.
BUG: refs.c:1119: create called without valid new_oid

Above is a copy of the command and reply.
This repository can be cloned with another computer and same command without problem using this version of git:

$ git --version --build-options
git version 2.27.0.windows.1
cpu: x86_64
built from commit: 907ab1011dce9112700498e034b974ba60f8b407
sizeof-long: 4
sizeof-size_t: 8
@derrickstolee
Copy link

Hi, @helarsen:

  1. Can you upgrade your Git version? 2.27.0 is quite old.

  2. Do you happen to have a publicly available repository that reproduces this bug? It's likely something to do with the ref names or references within that o_p_aux.git repository that is on your shared drive.

  3. If this reproduces on the latest Git for Windows version, then we will look into that BUG() statement.

There has been recent work in the refs backend that could have temporarily triggered this bug, but maybe it is fixed by now. If not, then we will take a closer look.

@helarsen
Copy link
Author

Thank you @derrickstolee for looking into this.

re 1: The version provoking this error is the latest 2.35.1.windows.2. And Version 2.27 is the one which does not fail (on another machine)
re 2: Unfortunately I dont have the repo in a public place. In fact it fails the same for two different repos (also on the file share) so it is less likely that both are broken. Both work on 2.27.
re 3: So it is failing on the latest version 2.35.

I should probably also have mentioned that despite this error - the cloned version does not appear to have any flaws - at least I can check in and out - very basic tests in that respect because of the obvious risk of corruption i abandoned working on the clone.

If i can have access to older releases of the installer I can try downgrade from 2.35 but I have not been able to find this - and I am reluctant to upgrade my working 2.27.

Can you point me where I can find older releases?
Can you suggest a git command to verify if the repo has a good health - as far as that goes?

br henning

@derrickstolee
Copy link

re 1: The version provoking this error is the latest 2.35.1.windows.2. And Version 2.27 is the one which does not fail (on another machine) re 2: Unfortunately I dont have the repo in a public place. In fact it fails the same for two different repos (also on the file share) so it is less likely that both are broken. Both work on 2.27. re 3: So it is failing on the latest version 2.35.

Sorry that I misread your issue. You clearly stated that and it was my fault in misunderstanding.

I should probably also have mentioned that despite this error - the cloned version does not appear to have any flaws - at least I can check in and out - very basic tests in that respect because of the obvious risk of corruption i abandoned working on the clone.

You could try a git fsck after cloning just to be sure.

If i can have access to older releases of the installer I can try downgrade from 2.35 but I have not been able to find this - and I am reluctant to upgrade my working 2.27.

Makes sense, but you're likely missing some security fixes in the meantime.

Can you point me where I can find older releases?

The full release list is available at https://github.com/git-for-windows/git/releases

Can you suggest a git command to verify if the repo has a good health - as far as that goes?

git fsck is the command you are looking for.

@helarsen
Copy link
Author

Thanks.
git fsck command was very useful.
I get these errors on the two master repositories:

heel@HPSXW1 MINGW64 /g/My Drive/Git Repos/o_p.git (BARE:master)
$ git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (562/562), done.
error: refs/desktop.ini: invalid sha1 pointer 0000000000000000000000000000000000000000
error: refs/heads/desktop.ini: invalid sha1 pointer 0000000000000000000000000000000000000000
error: refs/tags/desktop.ini: invalid sha1 pointer 0000000000000000000000000000000000000000
heel@2HPSXW1 MINGW64 /g/My Drive/Git Repos/o_p_aux.git (BARE:master)
$ git fsck
bad sha1 file: ./objects/00/desktop.ini
bad sha1 file: ./objects/01/desktop.ini6)
bad sha1 file: ./objects/05/desktop.ini
bad sha1 file: ./objects/18/desktop.ini56)
bad sha1 file: ./objects/1b/desktop.ini
...... removed a lot of similar lines
bad sha1 file: ./objects/f4/desktop.ini256)
bad sha1 file: ./objects/fd/desktop.ini
bad sha1 file: ./objects/fe/desktop.ini256)
bad sha1 file: ./objects/ff/desktop.ini
Checking object directories: 100% (256/256), done.
Checking objects: 100% (431/431), done.
error: refs/desktop.ini: invalid sha1 pointer 0000000000000000000000000000000000000000
error: refs/heads/desktop.ini: invalid sha1 pointer 0000000000000000000000000000000000000000
error: refs/tags/desktop.ini: invalid sha1 pointer 0000000000000000000000000000000000000000

It turns out that Google drive, which hosts the repository sprinkles each and every folder with a desktop.ini file!

desktop.ini -->
[.ShellClassInfo]
ConfirmFileOp=0
IconResource=C:\Program Files\Google\Drive File Stream\56.0.9.0\GoogleDriveFS.exe,23

So google drive it is not a good place to store such repositories!
I will move to another store and delete all those files.
By the way - some of the lines in the git fsck output are a bit odd:

bad sha1 file: ./objects/01/desktop.ini6)
bad sha1 file: ./objects/14/desktop.ini56)
bad sha1 file: ./objects/fe/desktop.ini256)

with a file extension of .ini6) or .ini56) or .ini256). This is not reflected in the actual files - they are all .ini

I will clean up this shameful mess and report back if this did not solve the issue.

Any suggestions are welcome - of course.

@helarsen
Copy link
Author

helarsen commented Apr 12, 2022

Summary and solutions.
Fail of cloning from a bare repository hosted on a google drive has been identified to the fact that google drive adds a desktop.ini file to all the folders. So:

Cloning fails in git 2.35 with the mentioned message "BUG: refs.c:1119: create called without valid new_oid" but
Cloning with git 2.27 does not show this message and everything seems to work.
git fsck reveals though that there are bad sha1 files for a large number of files named desktop.ini

My solution: Copy the repository files from google drive to a windows folder and remove all the desktop.ini files (using the practical grepWin tool). Now the repostory is healthy again in its new location.

But lesson is - "Dont store a repository on google drive".

Thanks to @derrickstolee for helping

@derrickstolee
Copy link

I'm glad you have a workaround. We should probably fix this, anyway. Having a BUG: error isn't good. It's likely that there was a missing error check somewhere higher up. You've given me enough of a repro to start investigating. Thanks!

derrickstolee added a commit to derrickstolee/git that referenced this issue Apr 13, 2022
When cloning directly from a local repository, we load a list of refs
based on scanning the $GIT_DIR/refs/ directory of the "server"
repository. If files exist in that directory that do not parse as
hexadecimal hashes, then the ref array used by write_remote_refs()
ends up with some entries with null OIDs. This causes us to hit a BUG()
statement in ref_transaction_create():

  BUG: create called without valid new_oid

This BUG() call used to be a die() until 033abf9 (Replace all
die("BUG: ...") calls by BUG() ones, 2018-05-02). Before that, the die()
was added by f04c5b5 (ref_transaction_create(): check that new_sha1 is
valid, 2015-02-17).

The original report for this bug [1] mentioned that this problem did not
exist in Git 2.27.0. The failure bisects unsurprisingly to 968f12f
(refs: turn on GIT_REF_PARANOIA by default, 2021-09-24). When
GIT_REF_PARANOIA is enabled, this case always fails as far back as I am
able to successfully compile and test the Git codebase.

[1] git-for-windows#3781

There are two approaches to consider here. One would be to remove this
BUG() statement in favor of returning with an error. There are only two
callers to ref_transaction_create(), so this would have a limited
impact.

The other approach would be to add special casing in 'git clone' to
avoid this faulty input to the method.

While I originally started with changing 'git clone', I decided that
modifying ref_transaction_create() was a more complete solution. This
prevents failing with a BUG() statement when we already have a good way
to report an error (including a reason for that error) within the
method. Both callers properly check the return value and die() with the
error message, so this is an appropriate direction.

The added test helps check against a regression, but does check that our
intended error message is handled correctly.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
derrickstolee added a commit to derrickstolee/git that referenced this issue Apr 25, 2022
When cloning directly from a local repository, we load a list of refs
based on scanning the $GIT_DIR/refs/ directory of the "server"
repository. If files exist in that directory that do not parse as
hexadecimal hashes, then the ref array used by write_remote_refs()
ends up with some entries with null OIDs. This causes us to hit a BUG()
statement in ref_transaction_create():

  BUG: create called without valid new_oid

This BUG() call used to be a die() until 033abf9 (Replace all
die("BUG: ...") calls by BUG() ones, 2018-05-02). Before that, the die()
was added by f04c5b5 (ref_transaction_create(): check that new_sha1 is
valid, 2015-02-17).

The original report for this bug [1] mentioned that this problem did not
exist in Git 2.27.0. The failure bisects unsurprisingly to 968f12f
(refs: turn on GIT_REF_PARANOIA by default, 2021-09-24). When
GIT_REF_PARANOIA is enabled, this case always fails as far back as I am
able to successfully compile and test the Git codebase.

[1] git-for-windows#3781

There are two approaches to consider here. One would be to remove this
BUG() statement in favor of returning with an error. There are only two
callers to ref_transaction_create(), so this would have a limited
impact.

The other approach would be to add special casing in 'git clone' to
avoid this faulty input to the method.

While I originally started with changing 'git clone', I decided that
modifying ref_transaction_create() was a more complete solution. This
prevents failing with a BUG() statement when we already have a good way
to report an error (including a reason for that error) within the
method. Both callers properly check the return value and die() with the
error message, so this is an appropriate direction.

The added test helps check against a regression, but does check that our
intended error message is handled correctly.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
gitster pushed a commit to git/git that referenced this issue Apr 25, 2022
When cloning directly from a local repository, we load a list of refs
based on scanning the $GIT_DIR/refs/ directory of the "server"
repository. If files exist in that directory that do not parse as
hexadecimal hashes, then the ref array used by write_remote_refs()
ends up with some entries with null OIDs. This causes us to hit a BUG()
statement in ref_transaction_create():

  BUG: create called without valid new_oid

This BUG() call used to be a die() until 033abf9 (Replace all
die("BUG: ...") calls by BUG() ones, 2018-05-02). Before that, the die()
was added by f04c5b5 (ref_transaction_create(): check that new_sha1 is
valid, 2015-02-17).

The original report for this bug [1] mentioned that this problem did not
exist in Git 2.27.0. The failure bisects unsurprisingly to 968f12f
(refs: turn on GIT_REF_PARANOIA by default, 2021-09-24). When
GIT_REF_PARANOIA is enabled, this case always fails as far back as I am
able to successfully compile and test the Git codebase.

[1] git-for-windows#3781

There are two approaches to consider here. One would be to remove this
BUG() statement in favor of returning with an error. There are only two
callers to ref_transaction_create(), so this would have a limited
impact.

The other approach would be to add special casing in 'git clone' to
avoid this faulty input to the method.

While I originally started with changing 'git clone', I decided that
modifying ref_transaction_create() was a more complete solution. This
prevents failing with a BUG() statement when we already have a good way
to report an error (including a reason for that error) within the
method. Both callers properly check the return value and die() with the
error message, so this is an appropriate direction.

The added test helps check against a regression, but does check that our
intended error message is handled correctly.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants