git reading mapped drive from eXist db: short read error (due to improperly reported file size?)

[Torcsi]

• I was not able to find an open or closed issue matching what I'm seeing

Also reported as eXist-db/exist#1701

Setup

$ git --version --build-options

git version 2.16.0.windows.2
cpu: x86_64
built from commit: a56c4f9e2a9513d2be57dc4346e0e84887399cd9
sizeof-long: 4

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.16299.192]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Notepad++
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

exist-db installation required to reproduce the issue
http://exist-db.org/exist/apps/homepage/index.html => download, install, admin/admin username password

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

cmd

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

net use p: http://localhost:8080/exist/webdav/db/ /user:admin admin
mkdir p:\apps\test
echo “my file” > p:\apps\test\index.html
P:
Cd \apps\test
mkdir c:\temp\test
git init git init --separate-git-dir C:\temp\test
git init add ./index.html

• What did you expect to occur after running these commands?

...file added to the repository...

• What actually happened instead?

error: short read while indexing index.html
error: index.html: failed to insert into database
error: unable to index file index.html
fatal: adding files failed

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

** insert URL here **

Possible reason
exist uses Milton.
exist WebDAV improperly reports size of file as found in http://exist-db.org/exist/apps/doc/repo/webdav.xml

“The size of an XML document is presented as a multiple of 4096 bytes, which is eXist's internal pagesize (see conf.xml). The actual size on an XML document stored in the database can not be determined because the size depends on many factors, e.g. the applied serialization parameters.”

As such, git will not get exact file size. When trying to read the entire file, it cannot read whole as expected from the file length, since the file size was just an estimation.

[dscho]

That is really unfortunate. Git relies on accurate file size information, it cannot function without it.

I guess there are only two ways out of this:

1. get eXist to fix this, or
2. stop using Git on drives mapped into eXist's database.

[Torcsi]

@dscho thanks; does git use lseek END, fstat or stat to determine length? sorry cannot dig into the code.
For eXist: stat seems nearly impossible to solve (XML content rendered during serialization, size is not known).
Btw. if anyway content is to be entirely read, isn't this a small file optimization to get the length?
I wish to know it for helping eXist maintainers, what to ask from them if that could be supported.

[dscho]

does git use lseek END, fstat or stat to determine length?

lstat(). Which in Git for Windows is either performed via GetFileAttributesW() (without the FSCache feature) or via FindFirstFileW()/FindNextFileW().

if anyway content is to be entirely read, isn't this a small file optimization to get the length?

But it is not! The entire idea of Git's index is to avoid reading the files' contents just to know whether they are up-to-date, and instead use the mtime and file size (and a couple more indicators for change).

It is a core requirement of Git to know accurate file sizes. It cannot work without.

[Torcsi]

Thanks. I have passed the information to eXist developers. There is very little chance that they can do anything with that, since size cannot be computed without traversing the full XML file. We must work with an extra staging folder...

As for entire read:
Clearly Git should/does not read the file when not needed - just realized from practice that files whose date was modified but the content was not, the file was not updated with the new date. I guess in this case the file had to be read.

Since this is not a GIT issue, I close this issue

Thank you for excellent support.