Add initial / PoC gittuf remote handling protocol

[adityasaky]

Edit 2024-05-24: I think the transport is at least at a proof of concept level now. Current status:

• HTTP(s) and SSH remotes
• Clone/fetch also updates refs/gittuf/ with what's on the remote
• Push creates signed RSL entries for each ref being pushed
• Clone has been tested across different repositories, including some "large" repositories such as the kernel

There are a bunch of TODOs and FIXMEs in the code. There's also a bunch of repeated code between the curl backend and the ssh backend that we can maybe clean up. Things that we also want to do:

• On push, first check local RSL is updated and then create RSL entries
• On push, support high traffic repositories where RSL entries may be pushed after they are fetched and before locally created entry is submitted
• gittuf must support a mismatch in remote and local refs #413
• On fetch, verify new RSL entries (there's some commented out code for this, we need some other changes)
• Support local file backend (when the "remote" is available locally)

Play with it:

• Fetch this branch
• Run make install (it should also place git-remote-gittuf in your GOBIN, so this assumes that's in your PATH)
• Clone using gittuf protocol; Eg: git clone gittuf::git@github.com:gittuf/gittuf for ssh, git clone gittuf::https://github.com/gittuf/gittuf for https
• Proceed with your standard git push/pull/fetch operations
• Ensure you have commit signing configured in Git config to sign RSL entries for pushes

h/t @SantiagoTorres for insights on how this works!

[adityasaky]

I think this is ready for a first round of feedback!

@wlynch @JustinCappos @lukpueh @patzielinski @neilnaveen

[adityasaky]

Note: as of 13e4a00, the transport doesn't work. I'm working on making it more resilient to different local conditions.

[adityasaky]

Things work again (and are more robust than before). I also tested out some trivial force pushes / fetches and they work as well, though we will need #369 in some form to mark rsl entries for rebased comments as skipped.

[neilnaveen]

A feature that would work well with the transport, since it controls all pushes and pulls, is caching the latest verified RSL entry in a specific reference, allowing only the user's signing key to commit to it. This approach would significantly speed up verification. After the initial pull from the main repository, gittuf's impact on the user experience would be minimal, even with large repositories, as long as users pull from their remote regularly.

[adityasaky]

Yeah, we definitely need that (#245). This and a couple other things make me wonder if we want to carve out refs/gittuf-local/ so that it doesn't get synced accidentally.

[adityasaky]

the code is still a mess and i want to work on cleaning things up, but the functionality seems fairly stable. This hasn't switched on gittuf verify-ref yet, I'm waiting on some other changes to land. In the meantime, seeking feedback and opinions on the best way to structure the curl and ssh helpers!

[lukpueh]

Impressive work, Aditya! Your code walkthrough the other day was very helpful, but I must admit, I still find it hard to navigate by myself.

Maybe you could try to better separate (1) main flow, (2) transport, and (3) gittuf logic.

IIUC (1) is controlled by the messages received from and sent to the git parent process, where stdin/stdout are the read and write message queues.

By (2) I mean the messages sent to and received from the http transport (curl) or ssh client (ssh) child processes spawned by us, with similar queues.

And finally, (3) would be the interposed gittuf logic this feature is actually about, i.e. fetching, verifying, creating and pushing additional refs.

It would be extremely helpful to see at one glance the expected messages in both directions and on both ends, and maybe even their typical sequence. If this is not possible in code, it could be done in documentation.

What could be done in code, would be to reduce the amount of nesting. In that regard, I have one specific question: Is the internal currentState variable necessary, or would the flow of messages be enough state, e.g. could the main loop have a single dispatch level based on the message content?

[adityasaky]

@lukpueh and I have discussed how to improve this package and make it more maintainable (see his comment above for the general improvements). I propose we do this iteratively, separately, and for now go ahead and merge this as an alpha implementation of the transport.

@wlynch @patzielinski @neilnaveen

[patzielinski]

Since this doesn't affect people who don't explicitly opt-in to using the transport, I think we can indeed go ahead and merge this in. I can open an issue about this being in alpha and needing improvements.

Side note: we should also probably add something to the docs discussing/describing/detailing this.

[neilnaveen]

This looks good. I haven't run it; I have just one suggestion. Thank you!

+1 on adding docs detailing that this is an alpha feature, and also explaining the current functionalities it adds on top of git, maybe somthing like a list of commands that have additional calls to gittuf, and what those calls are/do.

[neilnaveen]

Is this supposed to still be here, or is this something that should be uncommented later on?

[adityasaky]

the latter

[adityasaky]

Suggestions on the best place to put the docs? We don't get nice auto-gen in docs/cli like with the main cmd package.