-
Notifications
You must be signed in to change notification settings - Fork 25.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'jt/fetch-cdn-offload' into pu
WIP for allowing a response to "git fetch" to instruct the bulk of the pack contents to be instead taken from elsewhere (aka CDN). * jt/fetch-cdn-offload: SQUASH??? upload-pack: send part of packfile response as uri fetch-pack: support more than one pack lockfile upload-pack: refactor reading of pack-objects out Documentation: add Packfile URIs design doc Documentation: order protocol v2 sections http-fetch: support fetching packfiles by URL http: improve documentation of http_pack_request http: use --stdin when getting dumb HTTP pack
- Loading branch information
Showing
17 changed files
with
669 additions
and
132 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
Packfile URIs | ||
============= | ||
|
||
This feature allows servers to serve part of their packfile response as URIs. | ||
This allows server designs that improve scalability in bandwidth and CPU usage | ||
(for example, by serving some data through a CDN), and (in the future) provides | ||
some measure of resumability to clients. | ||
|
||
This feature is available only in protocol version 2. | ||
|
||
Protocol | ||
-------- | ||
|
||
The server advertises `packfile-uris`. | ||
|
||
If the client then communicates which protocols (HTTPS, etc.) it supports with | ||
a `packfile-uris` argument, the server MAY send a `packfile-uris` section | ||
directly before the `packfile` section (right after `wanted-refs` if it is | ||
sent) containing URIs of any of the given protocols. The URIs point to | ||
packfiles that use only features that the client has declared that it supports | ||
(e.g. ofs-delta and thin-pack). See protocol-v2.txt for the documentation of | ||
this section. | ||
|
||
Clients then should understand that the returned packfile could be incomplete, | ||
and that it needs to download all the given URIs before the fetch or clone is | ||
complete. | ||
|
||
Server design | ||
------------- | ||
|
||
The server can be trivially made compatible with the proposed protocol by | ||
having it advertise `packfile-uris`, tolerating the client sending | ||
`packfile-uris`, and never sending any `packfile-uris` section. But we should | ||
include some sort of non-trivial implementation in the Minimum Viable Product, | ||
at least so that we can test the client. | ||
|
||
This is the implementation: a feature, marked experimental, that allows the | ||
server to be configured by one or more `uploadpack.blobPackfileUri=<sha1> | ||
<uri>` entries. Whenever the list of objects to be sent is assembled, a blob | ||
with the given sha1 can be replaced by the given URI. This allows, for example, | ||
servers to delegate serving of large blobs to CDNs. | ||
|
||
Client design | ||
------------- | ||
|
||
While fetching, the client needs to remember the list of URIs and cannot | ||
declare that the fetch is complete until all URIs have been downloaded as | ||
packfiles. | ||
|
||
The division of work (initial fetch + additional URIs) introduces convenient | ||
points for resumption of an interrupted clone - such resumption can be done | ||
after the Minimum Viable Product (see "Future work"). | ||
|
||
The client can inhibit this feature (i.e. refrain from sending the | ||
`packfile-uris` parameter) by passing --no-packfile-uris to `git fetch`. | ||
|
||
Future work | ||
----------- | ||
|
||
The protocol design allows some evolution of the server and client without any | ||
need for protocol changes, so only a small-scoped design is included here to | ||
form the MVP. For example, the following can be done: | ||
|
||
* On the server, a long-running process that takes in entire requests and | ||
outputs a list of URIs and the corresponding inclusion and exclusion sets of | ||
objects. This allows, e.g., signed URIs to be used and packfiles for common | ||
requests to be cached. | ||
* On the client, resumption of clone. If a clone is interrupted, information | ||
could be recorded in the repository's config and a "clone-resume" command | ||
can resume the clone in progress. (Resumption of subsequent fetches is more | ||
difficult because that must deal with the user wanting to use the repository | ||
even after the fetch was interrupted.) | ||
|
||
There are some possible features that will require a change in protocol: | ||
|
||
* Additional HTTP headers (e.g. authentication) | ||
* Byte range support | ||
* Different file formats referenced by URIs (e.g. raw object) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.