Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clone failure on android.googlesource.com repos #2518

Closed
mduggan opened this issue Aug 15, 2014 · 4 comments · Fixed by #2525
Closed

Clone failure on android.googlesource.com repos #2518

mduggan opened this issue Aug 15, 2014 · 4 comments · Fixed by #2525

Comments

@mduggan
Copy link

mduggan commented Aug 15, 2014

Using libgit2 to clone any of the repositories here:
https://android.googlesource.com/
fails with Can't fit data in the buffer (at transports/http.c:L386):

mduggan@sparklemotion:~/src/libgit2/examples/network[master]$ ./git2 clone https://android.googlesource.com/device/sample test
net   0% (   3 kb,     0/ 1436)  /  idx   0% (    0/ 1436)  /  chk   0% (   0/   0) (null)

ERROR 12: Can't fit data in the buffer
Bad news:
 Can't fit data in the buffer

The buffer is transport_smart.buffer_data in transports/smart.h. I tried the naive solution and bumped the size of this buffer to 262144, which made the problem go away. I don't understand the protocol well enough to know what's wrong though - I expect there is a nicer fix than that?

@carlosmn
Copy link
Member

The size of that buffer is slightly larger than the maximum packet size as defined by the Git Smart Protocol[1] so there should never be an issue filling it. My first impression is that It looks like whatever is serving the android repositories might be trying to send more data than it's allowed, though it's odd that git wouldn't face the same issue.

Annoyingly, this host only seems to be available through HTTPS which makes capturing the output a PITA.

[1] https://github.com/git/git/blob/master/Documentation/technical/protocol-capabilities.txt#L118-L151

@carlosmn
Copy link
Member

The good new is that gerrit is not breaking the protocol.

The issue seems to be some bad combination of recv timing on our http parser side. The http parser pushes data into our buffer, and I'm seeing two successive pushes, one of which is just over the size we have left in our buffer.

It looks like we're receiving the end of one HTTP chunk which coincides with the end of the pkt, but our recv buffer has the start of the next chunk, so http-parser sends that as well, which would overflow our buffer.

Making the buffer larger is one way to make sure this doesn't happen, but it's probably not the most correct one. I think that if we make the recv buffer (parse_buffer) think it's full of data, it will only try to retrieve as much data as we could fit into the buffer and would then be guaranteed that the buffer would never overflow.

carlosmn added a commit that referenced this issue Aug 16, 2014
The recv buffer (parse_buffer) and the buffer have independent sizes and
offsets. We try to fill in parse_buffer as much as possible before
passing it to the http parser. This is fine most of the time, but fails
us when the buffer is almost full.

In those situations, parse_buffer can have more data than we would be
able to put into the buffer (which may be getting full if we're towards
the end of a data sideband packet).

To work around this, we check if the space we have left on our buffer is
smaller than what could come from the network. If this happens, we make
parse_buffer think that it has as much space left as our buffer, so it
won't try to retrieve more data than we can deal with.

As the start of the data may no longer be at the start of the buffer, we
need to keep track of where it really starts (data_offset) and use that
in our calculations for the real size of the data we received from the
network.

This fixes #2518.
@vmg vmg closed this as completed in #2525 Aug 18, 2014
carlosmn added a commit that referenced this issue Aug 18, 2014
The recv buffer (parse_buffer) and the buffer have independent sizes and
offsets. We try to fill in parse_buffer as much as possible before
passing it to the http parser. This is fine most of the time, but fails
us when the buffer is almost full.

In those situations, parse_buffer can have more data than we would be
able to put into the buffer (which may be getting full if we're towards
the end of a data sideband packet).

To work around this, we check if the space we have left on our buffer is
smaller than what could come from the network. If this happens, we make
parse_buffer think that it has as much space left as our buffer, so it
won't try to retrieve more data than we can deal with.

As the start of the data may no longer be at the start of the buffer, we
need to keep track of where it really starts (data_offset) and use that
in our calculations for the real size of the data we received from the
network.

This fixes #2518.
@sbc100
Copy link
Contributor

sbc100 commented Nov 17, 2014

Any chance we can merge this change in the maintenance release? I was hoping it would be v0.21.2. If there is going to be a v0.21.3 could this be included please? (Sorry couldn't find the policy of merging change into maintenance releases but this seem like a genuine bug that effects anybody pulling from googlesource repositories).

@carlosmn
Copy link
Member

This was included in the maintenance branch as 7d729d0 which is included in v0.21.2.

Schibum pushed a commit to Schibum/naclports that referenced this issue Jan 9, 2015
This fixes the issue of accessing googlesource.com repos:
libgit2/libgit2#2518

R=bradnelson@google.com

Review URL: https://codereview.chromium.org/744243002
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants