Support syncing to S3

[justinsb]

Initial support for kpromo to sync to S3 buckets.

Added initial support for kpromo file synchronization to S3 buckets.

[justaugustus]

@justinsb -- AFK for the moment, but can you check if it would instead make sense to implement portions of this within https://github.com/kubernetes-sigs/release-sdk/blob/main/object/store.go?

[justinsb]

@justaugustus I'm not sure that API is a great match here, as it looks like in order to copy files across providers we would need providers to know about each other. (We have an pretty broad vfs library here if you want generic functionality , I think the abstractions there are a bit more evolved.)

But for this PR, the real nuance is in carefully setting the metadata etc to make the common case fast (i.e. the case when nothing has to be copied). That's why we have to use a non-parallel object upload, so that the ETag is the MD5 hash. I think for integrity purposes we also want to upload additional hashes as metadata, but we're really very optimized for our particular use-case here.

I'm happy to evolve the API in release-sdk, but I'd probably steer it towards something with objects paths as first class concepts; WDYT about just sharing the vfs layer with kOps?

[justaugustus]

@justaugustus I'm not sure that API is a great match here, as it looks like in order to copy files across providers we would need providers to know about each other. (We have an pretty broad vfs library here if you want generic functionality , I think the abstractions there are a bit more evolved.)

TIL about https://github.com/kubernetes/kops/tree/master/util/pkg/vfs!

I'm happy to evolve the API in release-sdk, but I'd probably steer it towards something with objects paths as first class concepts; WDYT about just sharing the vfs layer with kOps?

@justinsb -- Happy for you to move forward using kOps vfs here, if it's going to dedupe some of the work.

Are changes to kops/util/pkg generally minimal across Kubernetes major versions? (My concern being tying the release engineering tooling to the k/k versioning complexities.)
A way to minimize this concern could be to donate vfs (and/or some larger subset of the util) package to sigs.k8s.io/release-sdk. Would that be something y'all would be open to?

[justinsb]

Definitely open to putting vfs somewhere shared :-) It's already shared between etcdadm and kOps, so it is generally useful. It's pretty stable although it's recently seen a flood of activity because we're adding context.Context as part of the great logging/tracing rewrite.

I was also thinking though that the API we have here is probably a reasonable minimal API, and we have higher level functionality (like folder-to-folder syncing) in kpromo. So we could also just move the filestores.

I think two good options are to put kOps VFS in a standalone repo somewhere (because it's not really only release tooling, either) or to sync kpromo's file abstractions with release-sdk (they are more logically coupled).

I would ask though if we can merge this PR in the interim though, it's all part of the S3 mirroring that reduces the CNCF expenditure on egress bandwidth - we finally have AWS buckets :-)

[justaugustus]

I would ask though if we can merge this PR in the interim though, it's all part of the S3 mirroring that reduces the CNCF expenditure on egress bandwidth - we finally have AWS buckets :-)

Yep, let's just import what makes sense here and minimize the pain!

[justinsb]

Thanks @justaugustus ... I was thinking that this and release are actually pretty specific functionality, for example I think we want to upload in a potentially suboptimal way to preserve metadata, we might want to force the metadata always to be created etc, so it belongs in a library that promo-tools and the rest of the release tooling share,it's not a generic VFS layer.

So I propose we merge this here, and then I'll submit some refactorings both to release-sdk and here so that we can have a shared library in release-sdk and use it here (i.e. I'll try upstream the code here and refactor release-sdk to avoid calling gsutil where it isn't applicable). I haven't poked around where all the places release-sdk is used yet though, not entirely sure what I'm signing up for yet :-)

[justaugustus]

@justinsb -- Left some nits. If you want to fix them here, go for it.

If you want to do these as a follow-up, feel free to release the hold.
/hold

[justaugustus]

Suggested change

	if strings.HasPrefix(filestore.Base, "gs://") {
	if strings.HasPrefix(filestore.Base, object.GcsPrefix) {

[justaugustus]

Same comment about defining the scheme as above.

[justaugustus]

Let's define something like:

var supportedURISchemes := []string{
  "s3",
  "gs",
}

And then:

	return nil, fmt.Errorf(
		"unrecognized scheme %q (supported schemes: %s)",
		filestore.Base,
                strings.Join(supportedURISchemes, ", "),
	)

[justinsb]

Created a register of providers, so that everything stays in sync here. I think that's the underlying concern, and I agree with it 👍

[justaugustus]

Same comment about object.GcsPrefix as above.

[justinsb]

Done - put these into the api, as I think they are our values, not the library's values (i.e. if the library changed the value, we would not want to change our values!)

[justaugustus]

Thanks @justinsb!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justaugustus]

/hold cancel