Emulator get stuck when the single transaction is not commited or rolled back

[pedrohml]

When using emulator, if the current/single transaction is not finalized (committed/rolled back) then all following operations are aborted and emulator throws an error saying The emulator only supports one transaction at a time.

Reproducibility:

1. Running emulator on docker: docker run -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator:latest
2. Start a transaction
3. Force the application to finish (not gracefully)

Suggestion:

Add support to safely rollback any ongoing transaction(s). So the application can always execute it when starts

[olavloite]

We would prefer not adding any custom APIs to the emulator that are not present in Cloud Spanner.

You can however achieve what you want using the following trick:

1. The emulator does not allow you to start more than one read/write transaction per database.
2. Starting a new read/write transaction on a different session on the same database will fail for the above reason.
3. Starting a new read/write transaction on the same session that already has a read/write transaction will however succeed, as this automatically aborts the running transaction.
4. You can then rollback the transaction that was started in step 3 to get a 'transaction-free' database on the emulator.

That means that a small utility that does the following will reset the emulator for you:

1. Loop through all the databases on the emulator.
2. Loop through all the sessions on that database.
3. Start a new read/write transaction on each session, and then rollback that transaction.

The following class implements a couple of utility methods that you can use to reset the emulator before running your tests (Note: This example is in Java. Please give me a ping if you use a different programming language and are not able to convert it to that language):

package com.google.cloud.spanner;

import com.google.api.gax.core.NoCredentialsProvider;
import com.google.api.gax.grpc.InstantiatingGrpcChannelProvider;
import com.google.cloud.spanner.admin.database.v1.DatabaseAdminClient;
import com.google.cloud.spanner.admin.database.v1.DatabaseAdminSettings;
import com.google.cloud.spanner.admin.instance.v1.InstanceAdminClient;
import com.google.cloud.spanner.admin.instance.v1.InstanceAdminSettings;
import com.google.cloud.spanner.v1.SpannerClient;
import com.google.cloud.spanner.v1.SpannerSettings;
import com.google.spanner.admin.database.v1.Database;
import com.google.spanner.admin.instance.v1.Instance;
import com.google.spanner.v1.BeginTransactionRequest;
import com.google.spanner.v1.RollbackRequest;
import com.google.spanner.v1.Session;
import com.google.spanner.v1.Transaction;
import com.google.spanner.v1.TransactionOptions;
import com.google.spanner.v1.TransactionOptions.ReadWrite;
import io.grpc.ManagedChannelBuilder;

public class EmulatorUtil {

  /** Removes all read/write transactions for all databases on the emulator for the given project. */
  public static void resetAllEmulatorTransactions(String project) throws Exception {
    try (InstanceAdminClient instanceAdminClient =
            InstanceAdminClient.create(
                InstanceAdminSettings.newBuilder()
                    .setCredentialsProvider(NoCredentialsProvider.create())
                    .setTransportChannelProvider(
                        InstantiatingGrpcChannelProvider.newBuilder()
                            .setEndpoint("localhost:9010")
                            .setChannelConfigurator(ManagedChannelBuilder::usePlaintext)
                            .build())
                    .build());
        DatabaseAdminClient databaseAdminClient =
            DatabaseAdminClient.create(
                DatabaseAdminSettings.newBuilder()
                    .setCredentialsProvider(NoCredentialsProvider.create())
                    .setTransportChannelProvider(
                        InstantiatingGrpcChannelProvider.newBuilder()
                            .setEndpoint("localhost:9010")
                            .setChannelConfigurator(ManagedChannelBuilder::usePlaintext)
                            .build())
                    .build())) {
      for (Instance instance :
          instanceAdminClient.listInstances(String.format("projects/%s", project)).iterateAll()) {
        for (Database database :
            databaseAdminClient.listDatabases(instance.getName()).iterateAll()) {
          resetEmulatorTransactions(database);
        }
      }
    }
  }

  /** Removes all read/write transactions for the given database on the emulator. */
  public static void resetEmulatorTransactions(Database database) throws Exception {
    try (SpannerClient spannerClient =
        SpannerClient.create(
            SpannerSettings.newBuilder()
                .setCredentialsProvider(NoCredentialsProvider.create())
                .setTransportChannelProvider(
                    InstantiatingGrpcChannelProvider.newBuilder()
                        .setEndpoint("localhost:9010")
                        .setChannelConfigurator(ManagedChannelBuilder::usePlaintext)
                        .build())
                .build())) {
      for (Session session : spannerClient.listSessions(database.getName()).iterateAll()) {
        Transaction transaction =
            spannerClient.beginTransaction(
                BeginTransactionRequest.newBuilder()
                    .setSession(session.getName())
                    .setOptions(
                        TransactionOptions.newBuilder()
                            .setReadWrite(ReadWrite.newBuilder().build())
                            .build())
                    .build());
        spannerClient.rollback(
            RollbackRequest.newBuilder()
                .setSession(session.getName())
                .setTransactionId(transaction.getId())
                .build());
      }
    }
  }
}

[olavloite]

@pedrohml Does the above-mentioned workaround solve your problem?

[olavloite]

Closing this issue as a workaround has been provided. Please feel free to reopen if this does not solve the problem for you.

[pedrohml]

Thanks for the response. I am currently using python sdk. Will try to adapt the workaround there.

[olavloite]

Thanks for the response. I am currently using python sdk. Will try to adapt the workaround there.

OK, let me know if you run into any problems with that.

[wray27]

@pedrohml any success in producing a workaround in python?

[abachman]

👋🏻 Hi!

I'm hitting the same wall but using the Ruby client library. This approach seems to be working for me, but takes a little library hacking.

First, I patched in a list_sessions method to the Google-built client Service class to get a list of sessions for a database:

module Google
  module Cloud
    module Spanner
      class Service
        # add a missing list_sessions method
        # @param database [String] in the form of a full Spanner identifier like
        #                          "project/.../instance/.../database/..."
        def list_sessions(database:, call_options: nil, token: nil, max: nil)
          opts = default_options call_options: call_options
          request = {
            database: database,
            page_size: max,
            page_token: token
          }
          paged_enum = service.list_sessions request, opts
          paged_enum.response
        end
      end
    end
  end
end

The API documentation is unclear with respect to how a session should be used to begin a new transaction. Reading the Ruby library source though, shows that the Session class has create_empty_transaction, which is worth a shot.

Here is a complete, standalone Ruby example that does the same thing that @olavloite demonstrated:

require "google/cloud/spanner"

PROJECT_ID = "test-project"
EMULATOR_HOST = "localhost:9010"

# patch the service to add a missing method
module Google
  module Cloud
    module Spanner
      class Service
        # add a missing list_sessions method
        # @param database [String] in the form of a full Spanner identifier like
        #                          "project/.../instance/.../database/..."
        def list_sessions(database:, call_options: nil, token: nil, max: nil)
          opts = default_options call_options: call_options
          request = {
            database: database,
            page_size: max,
            page_token: token
          }
          paged_enum = service.list_sessions request, opts
          paged_enum.response
        end
      end
    end
  end
end

module EmulatorUtil
  def self.reset_all_emulator_transactions(project_id, emulator_host)
    project = Google::Cloud::Spanner.new(
      project_id: project_id,
      emulator_host: emulator_host
    )

    project.instances.all do |instance|
      puts "instance: #{instance.path}"
      instance.databases.all do |database|
        puts "  database: #{database.path}"
        each_session_for_database(database) do |session|
          puts "    session: #{session.path}"
          tx = session.create_empty_transaction
          session.rollback tx.transaction_id
        rescue => e
          puts "    error resetting session: #{e.details}"
        end
      end
    end
  end

  def self.each_session_for_database(database)
    # patched method, paginated
    session_result = database.service.list_sessions(database: database.path)
    next_page_token = session_result.next_page_token

    loop do
      session_result.sessions.each do |grpc_session|
        yield Google::Cloud::Spanner::Session.new(grpc_session, database.service)
      end

      break if next_page_token.empty?

      session_result = database.service.list_sessions(database: database.path, token: next_page_token)
      next_page_token = session_result.next_page_token
    end
  end
end

EmulatorUtil.reset_all_emulator_transactions(PROJECT_ID, EMULATOR_HOST)

---

edited to add: forcing this situation is much easier. Running this script and killing it suddenly is enough for me to consistently dead-hang the emulator:

require "google/cloud/spanner"
project = Google::Cloud::Spanner.new(
  project_id: ENV["PROJECT_ID"],
  emulator_host: ENV["EMULATOR_HOST"]
)

# with database schema: 
# CREATE TABLE Customers (
#   Id STRING(32) NOT NULL
# ) PRIMARY KEY (CustomerId);
database = project.client(ENV["INSTANCE_ID"], ENV["DATABASE_ID"])
loop do
  database.transaction do |tx|
    tx.execute "INSERT INTO Customers (Id) VALUES ('#{SecureRandom.hex(6)}')"
    sleep Random.rand(0.5..1.0)
  end
end

And that script is (sadly) a pretty good metaphor for any given Rails test suite.

In other words, every test run that exits unexpectedly is a coin toss on whether it will trigger this situation.