cartservice - Spanner as database option

[mikrovvelle]

Background

Add support for Cloud Spanner, a Google cloud-native, horizontally-scaling database, as a backend for CartService.

Change Summary

• Add a SpannerCartStore.cs capable of using Cloud Spanner as a persistence layer.
• Add checks in CartServices's Startup.cs, allowing use of environmental variables to switch between using Redis and Spanner as the backend
• Add documentation of how to get the microservices demo running with Spanner
• Add DDL for single-table 'carts' database in Spanner
• Add workaround to Dockerfile, for libgr-pc_csharp_ext to work correctly (with link to github issue in comment)

Testing Procedure

Carry out instructions in docs/spanner.md, check that CartService functions as expected.

NOTE:

A Spanner-supporting CartService image needs to be added to the image repo used by this project (gcr.io/google-samples/microservices-demo/cartservice) in order for the instructions in docs/spanner.md to work. Until such an image is committed to the repo, you will need a workaround. You'll need to:

1. Have an image build repository available that you can push to, and which your Kubernetes cluster can pull from (e.g. europe-west3-docker.pkg.dev/my-image-repo/hipstershop). Set its address to an environmental variable REPO_PREFIX.
2. build your own cartservice image using podman build --tag ${REPO_PREFIX}/cartservice ./src/cartservice/src/.
3. push the image: podman push ${REPO_PREFIX}/cartservice:latest
4. edit ./release/kubernetes-manifests.yaml and change the image: line for cartservice (currently line 521) so that it points to the pushed cartservice image (e.g. image: europe-west3-docker.pkg.dev/my-image-repo/hipstershop/cartservice:latest)

[mathieu-benoit]

Really great, like discussed offline, thank you very much for bringing this new capability upstream to Online Boutique!

I just left my first pass of comments, will do another review later based on that.

[mathieu-benoit]

That's LGTM for me at this stage, thanks @mikrovvelle for taking into account my comments! Really great job here!

@NimJay, over to you :)

[NimJay]

Issue:
When I ran this command:

gcloud spanner instances create ${SPANNER_INSTANCE_NAME} \
    --description="online boutique backend" \
    --config="${SPANNER_REGION_CONFIG}" \
    --project=${PROJECT_ID} \
    --instance-type=free-instance

I got:

ERROR: (gcloud.spanner.instances.create) unrecognized arguments: --instance-type=free-instance (did you mean '--trace-token'?)

I don't see the --instance-type option documented for:

• gcloud compute instances create
• gcloud alpha compute instances create

So just the double-check, I ran:

gcloud components update

And the command worked.
I'll add a note about the gcloud components update command. :)

[NimJay]

✅ Note added:

Note: If you see an error related to the --instance-type flag being unrecognized, run gcloud components update.

[NimJay]

I've updated the example to "regional-us-east5" because that's where the free instance is available.

[NimJay]

Praise: Thanks for using a CartItems.ddl file here and gcloud to create the table!

[NimJay]

Thought:
This might be a waste of time, so let's ignore for now, but...
It looks like cloud.google.com samples use PascalCase (instead of camelCase) for column names.
It'd be ideal if there was a convention set by Google that we follow here.

[NimJay]

I add

sed -i "s/cartservice:v0.4.0/cartservice:v0.4.0-spanner/g" ./release/updated-manifests.yaml

I will be building and pushing a Spanner-supporting image at gcr.io/google-samples/microservices-demo/cartservice:v0.4.0-spanner.
Eventually, when we create the next release (e.g., 0.4.1), that version's cartservice image will support Cloud Spanner by default (i.e., without the -spanner suffix).
So the above sed command I added is only relevant to anyone trying spanner.md with version 0.4.0.

If you see problems with this approach, please let me know. :)

[NimJay]

✅ Just pushed an image to gcr.io/google-samples/microservices-demo/cartservice:v0.4.0-spanner.

[NimJay]

Praise: Thanks, Daniel, for adding this bash comments! They help. :)

[NimJay]

Issue:

When I deployed Online Boutique with Cloud Spanner by following the instructions here and using a Cloud-Spanner supporting image, I saw the following errors on Online Boutique's front-end:

rpc error: code = FailedPrecondition desc = Can't access cart storage at projects/nimjay-ob-spanner-pr/instances/onlineboutique/databases/carts. System.ArgumentException: Format of the initialization string does not conform to specification starting at index 0.
   at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String, Int32, StringBuilder, Boolean, String& , String& )
   at System.Data.Common.DbConnectionOptions.ParseInternal(Dictionary`2, String, Boolean, Dictionary`2, Boolean)
   at System.Data.Common.DbConnectionOptions..ctor(String, Dictionary`2, Boolean)
   at System.Data.Common.DbConnectionStringBuilder.set_ConnectionString(String value)
   at cartservice.cartstore.SpannerCartStore.GetCartAsync(String) in /app/cartstore/SpannerCartStore.cs:line 139
could not retrieve cart
main.(*frontendServer).viewCartHandler
	/src/handlers.go:255
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2109
github.com/gorilla/mux.(*Router).ServeHTTP
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
main.(*logHandler).ServeHTTP
	/src/middleware.go:82
main.ensureSessionID.func1
	/src/middleware.go:109
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2109
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2947
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1991
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594

and

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.108.9.216:7070: connect: connection refused"
could not retrieve cart
main.(*frontendServer).viewCartHandler
	/src/handlers.go:255
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2109
github.com/gorilla/mux.(*Router).ServeHTTP
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
main.(*logHandler).ServeHTTP
	/src/middleware.go:82
main.ensureSessionID.func1
	/src/middleware.go:109
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2109
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2947
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1991
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594

It's been about 20 minutes since I created my Cloud Spanner instance.
Do you have any idea what might be causing this?

[NimJay]

For clarity, here are some screenshots:

Screenshot 1

Screenshot 2

[NimJay]

I think I figure it out: I think it's because I didn't specific --project=${PROJECT_ID} in step 4. 😅
Let me retry. :)

[NimJay]

Update: Still getting the same errors.

[mikrovvelle]

I'm getting the error with the image at gcr.io/google-samples/microservices-demo/cartservice:v0.4.0-spanner, but not with the one in my private repo. I'll try rebuilding mine and see if that can reproduce the issue.

[NimJay]

(Please don't feel obligated to respond outside your work hours! :))

Just leaving an update:
I added "Data Source=" to the connectionString (based on this documentation). See commit 8b78843.

Now I see the following error:

rpc error: code = FailedPrecondition desc = Can't access cart storage at Data Source=projects/nimjay-ob-spanner-pr/instances/onlineboutique/databases/carts. Google.Cloud.Spanner.Data.SpannerException: Internal error.
 ---> Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. TokenResponseException: Error:"Server response does not contain a JSON object. Status code is: BadRequest", Description:"", Uri:""", DebugException="Google.Apis.Auth.OAuth2.Responses.TokenResponseException: Error:"Server response does not contain a JSON object. Status code is: BadRequest", Description:"", Uri:""
   at Google.Apis.Auth.OAuth2.Responses.TokenResponse.FromHttpResponseAsync(HttpResponseMessage, IClock, ILogger)
   at Google.Apis.Auth.OAuth2.ComputeCredential.RequestAccessTokenAsync(CancellationToken)
   at Google.Apis.Auth.OAuth2.TokenRefreshManager.RefreshTokenAsync()
   at Google.Apis.Auth.OAuth2.TokenRefreshManager.GetAccessTokenForRequestAsync(CancellationToken)
   at Google.Apis.Auth.OAuth2.ServiceCredential.GetAccessTokenWithHeadersForRequestAsync(String , CancellationToken )
   at Grpc.Auth.GoogleAuthInterceptors.<>c__DisplayClass3_0.<<FromCredential>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Grpc.Net.Client.Internal.GrpcProtocolHelpers.ReadCredentialMetadata(DefaultCallCredentialsConfigurator, GrpcChannel, HttpRequestMessage, IMethod, CallCredentials)
   at Grpc.Net.Client.Internal.GrpcCall`2.ReadCredentials(HttpRequestMessage)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage, Nullable`1)")
   at Google.Api.Gax.Grpc.Gcp.GcpCallInvoker.<>c__DisplayClass24_0`2.<<AsyncUnaryCall>g__PostProcessPropagateResult|3>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Cloud.Spanner.V1.SessionPool.TargetedSessionPool.CreatePooledSessionsAsync(CancellationToken)
   at Google.Cloud.Spanner.V1.SessionPool.TargetedSessionPool.<>c__DisplayClass43_0.<<GetNursePoolBackToHealthTask>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Cloud.Spanner.V1.SessionPool.TargetedSessionPool.<GetNursePoolBackToHealthTask>g__WithCancellationTokenAsync|43_1[TTaskResult](Task`1, CancellationToken)
   at Google.Cloud.Spanner.V1.SessionPool.TargetedSessionPool.<>c__DisplayClass31_0.<<GetSessionAcquisitionTask>g__NurseAndRetryAsync|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Cloud.Spanner.V1.SessionPool.TargetedSessionPool.AcquireSessionImplAsync(TransactionOptions, CancellationToken)
   at Google.Cloud.Spanner.V1.SessionPool.TargetedSessionPool.AcquireSessionAsync(TransactionOptions, CancellationToken)
   at Google.Cloud.Spanner.Data.EphemeralTransaction.<>c__DisplayClass7_0.<<ExecuteReadOrQueryAsync>g__Impl|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Cloud.Spanner.Data.ExecuteHelper.WithErrorTranslationAndProfiling[T](Func`1, String, Logger)
   --- End of inner exception stack trace ---
   at Google.Cloud.Spanner.Data.ExecuteHelper.WithErrorTranslationAndProfiling[T](Func`1, String, Logger)
   at Google.Cloud.Spanner.Data.SpannerCommand.ExecutableCommand.ExecuteReaderAsync(CommandBehavior, TimestampBound, CancellationToken)
   at Google.Cloud.Spanner.Data.SpannerCommand.ExecutableCommand.ExecuteDbDataReaderAsync(CommandBehavior, TimestampBound, CancellationToken)
   at Google.Cloud.Spanner.Data.SpannerCommand.ExecuteReaderAsync(CommandBehavior, CancellationToken)
   at cartservice.cartstore.SpannerCartStore.GetCartAsync(String) in /app/cartstore/SpannerCartStore.cs:line 116
could not retrieve cart
main.(*frontendServer).homeHandler
	/src/handlers.go:69
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2109
github.com/gorilla/mux.(*Router).ServeHTTP
	/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
main.(*logHandler).ServeHTTP
	/src/middleware.go:82
main.ensureSessionID.func1
	/src/middleware.go:109
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2109
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2947
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1991
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594

[NimJay]

I reran the commands from step 4 (related to service accounts). The above error disappeared. Online Boutique with Spanner is now working for me! 👍

[NimJay]

Thank you, once again, Daniel (@mikrovvelle), for implementing this!

It worked on my own GCP project. See screenshot below.

And thank you, @mathieu-benoit, for the review (and connecting the dots with this effort).

Approved. 😄

[NimJay]

Merging.
All checks are passing except the snippet-bot-check. But that isn't relevant here anyway because we're not modifying any region tags.