Cloud Storage UrlSigner support for HMAC sigs

[NinoFloris]

Looking into the UrlSigner I noticed it specifically hardcodes the GOOG4-RSA-SHA256 algo. Could an extension to the UrlSigner be made to support HMAC signing?

The reason for this ask is that RSA signing is rather computationally expensive at the amount of signatures we create per request and we would like to switch to HMAC, ideally without having to maintain the requisite hmac secret derivation and signing bits.

[jskeet]

I've asked a colleague who knows more about what the Storage service supports than I do. (I see in the documentation that there's support for HMAC, but I'm not sure of the implications of that in terms of client libraries.)

[NinoFloris]

Thanks!

I found these two pages to give a decent overview
https://cloud.google.com/storage/docs/authentication/signatures
https://cloud.google.com/storage/docs/migrating#authentication

it looks a lot like the rsa signing in terms of query params and payload.

[jskeet]

There's discussion about this internally now. Just to set expectations, if we decide to do this, we'll want to do it across most/all of the languages for which we have client libraries - which means it's unlikely to be a quick fix, if you see what I mean.

[NinoFloris]

Meant to reply here earlier. Thanks for bringing it up and letting me know. We can wait, it's just something on our radar to improve when we can.

[jskeet]

Thanks - there's been some progress on our side, in terms of discussions. Nothing firm yet, but I'm going to investigate whether I can provide you with a workaround to implement it reasonably simply within your own code without a library change (but still using UrlSigner). I hope to get to that next week. (It may prove a dead-end, and would only be a temporary workaround anyway, but it would be better than nothing.)

[jskeet]

Okay, I've looked into this a bit further. There's good news and bad news. The good news is that I have managed to get this to work. The bad news is that there are two reasons it won't work without further changes to the library:

• The V4Signer always uses an algorithm value of "GOOG4-RSA-SHA256" in the request, whereas for HMAC we need "GOOG4-HMAC-SHA256"
• The blob signing stage needs to know the date field in the request. While we can fetch "the current date (UTC)" again, it does mean there's a race condition: around midnight we could end up creating a request with a date field of (say) 2021-03-29, and then sign it using a date of 2021-03-30 which would obviously be invalid

So there's no interim workaround - we'll need to wait for this to be prioritized appropriately and designed across multiple languages.

Here's the HMAC blob signer code though, mostly for my own reuse later on:

// Copyright 2021 Google LLC
// 
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// 
//     https://www.apache.org/licenses/LICENSE-2.0
// 
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using Google.Api.Gax;
using System;
using System.Globalization;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace Google.Cloud.Storage.V1
{
    public sealed partial class UrlSigner
    {
        /// <summary>
        /// Blob signer to implement signing using an HMAC secret.
        /// </summary>
        internal sealed class HmacBlobSigner : IBlobSigner
        {
            private const string Location = "auto";
            private const string Service = "storage";
            private const string Prefix = "GOOG4";
            private const string RequestType = "goog4_request";

            private readonly string _accessId;
            private readonly string _secret;
            private readonly IClock _clock;

            internal HmacBlobSigner(string accessId, string secret, IClock clock) =>
                (_accessId, _secret, _clock) = (accessId, secret, clock);

            public string Id => _accessId;

            public string CreateSignature(byte[] data)
            {
                // TODO: Fix the timing race condition here. It's just possible that 
                // the clock has ticked over from one day to the next. We really want
                // to use the same date that we used to create the request.
                string date = _clock.GetCurrentDateTimeUtc().ToString("yyyyMMdd", CultureInfo.InvariantCulture);

                // Implements the steps in https://cloud.google.com/storage/docs/authentication/signatures#derive-key
                var utf8 = Encoding.UTF8;
                byte[] initialKey = utf8.GetBytes(Prefix + _secret);
                byte[] keyDate = Hash(initialKey, utf8.GetBytes(date));
                byte[] keyRegion = Hash(keyDate, utf8.GetBytes(Location));
                byte[] keyService = Hash(keyRegion, utf8.GetBytes(Service));
                byte[] signingKey = Hash(keyService, utf8.GetBytes(RequestType));
                byte[] signature = Hash(signingKey, data);
                return Convert.ToBase64String(signature);

                static byte[] Hash(byte[] key, byte[] data)
                {
                    using (HMACSHA256 hmac = new HMACSHA256(key))
                    {
                        return hmac.ComputeHash(data);
                    }
                }
            }

            public Task<string> CreateSignatureAsync(byte[] data, CancellationToken cancellationToken) =>
                Task.FromResult(CreateSignature(data));
        }
    }
}