Improve Table Service Performance

[edwin-huber]

I am actively looking at options to improve performance.
The best option I have found until we check in a change for this is to change the the constructor for LokiTableMetadataStore in src\table\persistence\LokiTableMetadataStore.ts

You have 2 options.

Turn off persistence:

  public constructor(public readonly lokiDBPath: string) {
       this.db = new Loki(lokiDBPath, {
    });
  }

Enable the lfsa adapter, which creates files per partition:

  public constructor(public readonly lokiDBPath: string) {
    const lfsa = require("../../../node_modules/lokijs/src/loki-fs-structured-adapter.js");
    const lfsaAdapter = new lfsa();
    this.db = new Loki(lokiDBPath, {
      autosave: true,
      adapter: lfsaAdapter
      autosaveInterval: 5000
    });
  }

Addresses #1198

[edwin-huber]

@XiaoningLiu - We need to agree on implementing control schema compatible with VS Code and Visual Studio 2022.

[edwin-huber]

Dependency on #1061

[joelverhagen]

@edwin-huber, from my testing Table didn't get faster in-memory. The slow path for blob and queue was writing extents which is not done at all in tables. LokiJS does not persist immediately upon write so Loki + Table doesn't have disk in the request context, just in a background persist every 5 seconds. Perhaps I am misunderstanding the original issue however.

[joelverhagen]

I wrote a blog about my change which describes the perf change: https://www.joelverhagen.com/blog/2023/11/azurite-in-memory

[edwin-huber]

a background persist every 5 seconds.

It was exactly this IO causing systems to hang, writing large amounts of table data to disk...

[joelverhagen]

Makes sense, thanks for clarifying this for me !