This is based on the recent master batch that included ds/bundle-uri-....

Now that git clone --bundle-uri can download a bundle list from a plaintex
file in config format, we can use the same set of key-value pairs to
advertise a bundle list over protocol v2. At the end of this series:

 1. A server can advertise bundles when uploadPack.advertiseBundleURIs is
    enabled. The bundle list comes from the server's local config,
    specifically the bundle.* namespace.
 2. A client can notice a server's bundle-uri advertisement and request the
    bundle list if transfer.bundleURI is enabled. The bundles are downloaded
    as if the list was advertised from the --bundle-uri option.

Many patches in this series were adapted from Ævar's v2 RFC [1]. He is
retained as author and I added myself as co-author only if the modifications
were significant.

[1]
https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/

 * Patches 1-7 are mostly taken from [1], again with mostly minor updates.
   The one major difference is the packet line format being a single
   key=value format instead of a sequence of pairs. (In v3, these commits
   are significantly reorganized from [1].)

 * Patches 8-11 finish off the ability for the client to notice the
   capability, request the values, and download bundles before continuing
   with the rest of the download.

One thing that is not handled here but could be handled in a future change
is to disconnect from the origin Git server while downloading the bundle
URIs, then reconnecting afterwards. This does not make any difference for
HTTPS, but SSH may benefit from the reduced connection time. The git clone
--bundle-uri option did not suffer from this because the bundles are
downloaded before the server connection begins.

After this series, there is one more before the original scope of the plan
is complete: using creation tokens as a heuristic. See [2] for the RFC
version of those patches.

[2] https://github.com/derrickstolee/git/pull/22

Updates in v3
=============

Most of these updates are due to Victoria's very thorough review. Thanks!

 * What was patch 2 was split to be better understood.
 * The new patch 2 is only the new test script infrastructure for testing
   whether or not the server provides the bundle-uri capability. It is
   extended with other more complicated examples in later patches. The name
   was rewritten from lib-t5730-*.sh to lib-bundle-uri-protocol.sh and the
   variable names are renamed with the BUNDLE_URI_ prefix.
 * The new patch 3 performs the basic client interaction with the
   'bundle-uri' command, while still not being fully wired up on the server
   side. The tests do check that the client requests the bundle-uri command
   after seeing it in the server's capabilities. One important difference
   from earlier is that the check for server_supports_v2() was moved into
   the get_bundle_uri() method (underneath the vtable) because we need to
   check the handshake before calling that method. It makes most sense to
   put the handshake call there, so do it from the start.
 * Patch 4 carefully tests how the transfer.bundleURI config blocks the
   client-side request of the bundle-uri command. Later tests will use the
   GIT_TEST_BUNDLE_URI environment variable instead.
 * The new Patch 5 renames got_remote_heads to finished_handshake in 'struct
   git_transport_data' and that's it. That new value is then used in patch 6
   to indicate if we need to request the handshake in the bundle URI logic.
 * Patch 6 creates the ls-remote helper in 'test-tool bundle-uri' as before,
   but now only makes use of the finished_handshake member instead of
   creating a new one. The test helper represents an example consumer of
   transport_get_remote_bundle_uri() without first doing the server-side
   handshake, which motivates several of the placements of code within that
   method and get_bundle_uri() earlier in the series. The "quiet" option is
   also removed to simplify the test helper and to always communicate the
   inner errors to the user.
 * Patch 7 adds the server-side listing of bundle.* config values. The test
   scripts around these config values have been cleaned up since the
   previous version.
 * Patch 8 has another iteration of strbuf_strip_file_from_path() taking the
   feedback from Victoria and Ævar.
 * Patch 9 adds the relative path logic. The definition of the base path is
   clarified in the commit message and comments. An additional test shows
   what happens if the server advertises too many parent paths
   (unfortunately, a die(), and this is marked for cleanup later).
 * Patch 10 is identical to the old patch 8.
 * Patch 11 completes the work by having the client download the bundles
   provided by the server list. It fixes an if/else that should have been an
   if/else-if. A new test checks that the --bundle-uri=X option overrides
   the server advertisement.

Updates in v2
=============

 * Commit messages now refer to protocol v2 "commands" not "verbs".
 * Several edits were made to gitprotocol-v2.txt thanks to Victoria's
   thorough review.
 * strbuf_parent_directory() is renamed strbuf_strip_file_from_path() to
   make it more clear how it behaves when ending with a slash.

Thanks,

 * Stolee

Derrick Stolee (6):
  transport: rename got_remote_heads
  bundle-uri: serve bundle.* keys from config
  strbuf: introduce strbuf_strip_file_from_path()
  bundle-uri: allow relative URLs in bundle lists
  bundle-uri: download bundles from an advertised list
  clone: unbundle the advertised bundles

Ævar Arnfjörð Bjarmason (5):
  protocol v2: add server-side "bundle-uri" skeleton
  t: create test harness for 'bundle-uri' command
  clone: request the 'bundle-uri' command when available
  bundle-uri client: add boolean transfer.bundleURI setting
  bundle-uri client: add helper for testing server

 Documentation/config/transfer.txt      |   6 +
 Documentation/gitprotocol-v2.txt       | 201 +++++++++++++++++++++++
 builtin/clone.c                        |  22 +++
 bundle-uri.c                           |  87 +++++++++-
 bundle-uri.h                           |  32 ++++
 connect.c                              |  44 +++++
 remote.h                               |   5 +
 serve.c                                |   6 +
 strbuf.c                               |   6 +
 strbuf.h                               |  11 ++
 t/helper/test-bundle-uri.c             |  48 ++++++
 t/lib-bundle-uri-protocol.sh           | 212 +++++++++++++++++++++++++
 t/t5601-clone.sh                       |  59 +++++++
 t/t5701-git-serve.sh                   |  40 ++++-
 t/t5730-protocol-v2-bundle-uri-file.sh |  17 ++
 t/t5731-protocol-v2-bundle-uri-git.sh  |  17 ++
 t/t5732-protocol-v2-bundle-uri-http.sh |  17 ++
 t/t5750-bundle-uri-parse.sh            |  82 ++++++++++
 t/t9119-git-svn-info.sh                |   2 +-
 t/test-lib-functions.sh                |   7 +
 transport-helper.c                     |  13 ++
 transport-internal.h                   |   7 +
 transport.c                            |  88 ++++++++--
 transport.h                            |  19 +++
 24 files changed, 1036 insertions(+), 12 deletions(-)
 create mode 100644 t/lib-bundle-uri-protocol.sh
 create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh
 create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh
 create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh

base-commit: c03801e19cb8ab36e9c0d17ff3d5e0c3b0f24193

Submitted-As: https://lore.kernel.org/git/pull.1400.v3.git.1670262639.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.1400.git.1667264854.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.1400.v2.git.1668628302.gitgitgadget@gmail.com