Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading from a non-existent database causes memory usage to grow #1089

Closed
sebright opened this issue Apr 23, 2021 · 0 comments · Fixed by #1093
Closed

Reading from a non-existent database causes memory usage to grow #1089

sebright opened this issue Apr 23, 2021 · 0 comments · Fixed by #1093
Assignees
Labels
api: spanner triage me

Comments

@sebright
Copy link
Contributor

@sebright sebright commented Apr 23, 2021

It seems like requests that fail with a NOT_FOUND error can cause the Spanner client to consume memory. I suspect that it is caused by the invalidatedDbClients field in SpannerImpl:

private final List<DatabaseClientImpl> invalidatedDbClients = new ArrayList<>();

I ran this test program to reproduce the issue with Java 11 and google-cloud-spanner 6.3.2:

import com.google.cloud.spanner.DatabaseId;
import com.google.cloud.spanner.InstanceConfigId;
import com.google.cloud.spanner.InstanceId;
import com.google.cloud.spanner.InstanceInfo;
import com.google.cloud.spanner.Key;
import com.google.cloud.spanner.Spanner;
import com.google.cloud.spanner.SpannerOptions;
import java.util.Arrays;
import org.junit.Test;

public class FailedRequestMemoryUsageTest {

  @Test
  public void testFailedRequestMemoryUsage() throws Exception {
    Spanner spanner = SpannerOptions.getDefaultInstance().getService();
    String projectId = spanner.getOptions().getProjectId();
    InstanceId instanceId = InstanceId.of(projectId, "my-instance");
    InstanceConfigId instanceConfigId = InstanceConfigId.of(projectId, "regional-us-central1");
    spanner
        .getInstanceAdminClient()
        .createInstance(
            InstanceInfo.newBuilder(instanceId)
                .setInstanceConfigId(instanceConfigId)
                .setDisplayName("my-instance")
                .setNodeCount(1)
                .build())
        .get();
    DatabaseId databaseId = DatabaseId.of(instanceId, "my-database");

    for (int i = 0; i <= 10000; i++) {
      if (i % 500 == 0) {
        Runtime.getRuntime().gc();
        System.out.println(
            i + "\t" + (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()));
      }
      try {
        spanner
            .getDatabaseClient(databaseId)
            .singleUse()
            .readRow("MyTable", Key.of(0), Arrays.asList("MyColumn"));
      } catch (Exception e) {
        // Ignore NOT_FOUND error
      }
    }
  }
}
0	19573984
500	31193384
1000	35718312
1500	40337512
2000	44895408
2500	49444864
3000	54109824
3500	58690984
4000	63295664
4500	68078928
5000	72855264
5500	77463136
6000	82355544
6500	86576208
7000	91327472
7500	95821992
8000	100438752
8500	105055256
9000	109650504
9500	114148912
10000	118900040

java-spanner-memory-usage

@product-auto-label product-auto-label bot added the api: spanner label Apr 23, 2021
@olavloite olavloite self-assigned this Apr 23, 2021
olavloite added a commit that referenced this issue Apr 23, 2021
SpannerImpl would keep a reference to all database clients that had been
created and then invalidated because the database was deleted or did not
exists in the first place. If the getDatabaseClient(String) method would
be called multiple times for the same invalid database, the memory usage
of the application would continue to increase.

Fixes #1089
@yoshi-automation yoshi-automation added the triage me label Apr 25, 2021
thiagotnunes pushed a commit that referenced this issue Apr 25, 2021
SpannerImpl would keep a reference to all database clients that had been
created and then invalidated because the database was deleted or did not
exists in the first place. If the getDatabaseClient(String) method would
be called multiple times for the same invalid database, the memory usage
of the application would continue to increase.

Fixes #1089
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner triage me
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants