Document s3 substitutions

[graham-at-target]

Ideally this will be backported to 2.0.x.

Ideally we could get an authoritative, minimal list of privileges required but it seems most people testing grant s3:*.

cc @shlevy

[copumpkin]

Docs seem good & taught me stuff I didn't know about it, even though I use authenticated Nix S3 reads and writes dozens of times a day 😄

[copumpkin]

I suppose this is why it doesn't need s3:GetBucketLocation on reads? Seems like a problem 😦

[copumpkin]

Why is the behavior so different between reads and writes?

[graham-at-target]

In short...

Download code:

nix/src/libstore/download.cc

Lines 600 to 606 in 45bcf54

	#ifdef ENABLE_S3
	S3Helper s3Helper("", Aws::Region::US_EAST_1); // FIXME: make configurable
	auto slash = request.uri.find('/', 5);
	if (slash == std::string::npos)
	throw nix::Error("bad S3 URI '%s'", request.uri);
	std::string bucketName(request.uri, 5, slash - 5);
	std::string key(request.uri, slash + 1);

Upload code:

nix/src/libstore/s3-binary-cache-store.cc

Line 189 in 45bcf54

, s3Helper(profile, region)

The Upload code does a fancier arg parsing, which happens in a separate function:

nix/src/libstore/store-api.cc

Lines 838 to 851 in 45bcf54

	ref<Store> openStore(const std::string & uri_,
	const Store::Params & extraParams)
	{
	auto uri(uri_);
	Store::Params params(extraParams);
	auto q = uri.find('?');
	if (q != std::string::npos) {
	for (auto s : tokenizeString<Strings>(uri.substr(q + 1), "&")) {
	auto e = s.find('=');
	if (e != std::string::npos)
	params[s.substr(0, e)] = s.substr(e + 1);
	}
	uri = uri_.substr(0, q);
	}

and my C++-foo isn't strong enough with the time I have to rectify it :(.

[AmineChikhaoui]

I think this depends on the region also ? https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

[graham-at-target]

No, DNS is sufficient:

In a virtual-hosted–style URL, you can use either of these endpoints. If you make a request to the http://bucket.s3.amazonaws.com endpoint, the DNS has sufficient information to route your request directly to the Region where your bucket resides.

[AmineChikhaoui]

oh good to know :)