Improve large expansion sets

[goetzrrGit]

• Tickets addressed: Sequencing Server crashes creating a large expansion set #1025
• Review: By commit
• Merge strategy: Merge (no squash)

Description

Part 1 of 2 for #1025

This PR addresses a stability issue where creating large expansion sets caused server crashes due to excessive resource consumption by worker processes. Here's a summary of the improvements:

Problem:

• Compiling individual authored logic large expansion sets led to high memory usage by worker processes.
• Queuing multiple compilation jobs per worker exacerbated the issue, contributing to server crashes.
• Workers had not cap on resources

Solution:

• Chunking: Implemented chunking to distribute authored logic compilation tasks among a subset of workers, preventing resource overload on individual processes.
• Caching: Introduced a TS transpilation cache to eliminate repetitive compilations for the same logic units, significantly reducing memory usage and avoiding unnecessary worker load.
• Exposing worker properties to fine-tune workers per project

By default, I am spinning up 8 workers and giving them 1GB of heap.

Verification

I was able to create an expansion set with 73 authored logic without a crash. Before the fix 20+ would crash the server.

Future work

Implement a single background worker to transpile the authoring logic while the server is idle and cache the results. Help with the huge upfront cost .ex 70+ expansion logic takes about 13 minutes. This is cpu bounded so a beefier computer would help.

[dyst5422]

I like the solution here with the caching.

I had a similar need recently for throttling the number of inflight promises and came up with this solution which might be inspiration for a less intrusive solution to the batching. Benefit here is that it allows us to limit the number of in process workers, BUT doesn't block on all of them completing. So we can ensure the workers are all always filled.

/** Throttle number of in flight promises. */
export class InFlightPromiseThrottler {
  private inFlightPromises: Promise<unknown>[] = [];
  private promiseLimit: number;

  public constructor(promiseLimit: number) {
    this.promiseLimit = promiseLimit;
  }

  public async run<T>(promiseFactory: () => Promise<T>): Promise<T> {
    while (this.inFlightPromises.length >= this.promiseLimit) {
      await Promise.race(this.inFlightPromises);
    }
    const promise = promiseFactory();
    this.inFlightPromises.push(promise);
    return promise.finally(() => {
      const index = this.inFlightPromises.indexOf(promise);
      if (index !== -1) {
        this.inFlightPromises.splice(index, 1);
      }
    });
  }
}

Used like

// Somewhere in a higher scope
const workerPromiseThrottler = new InFlightPromiseThrottler(8);

// When we actually do the worker run

const result = await workerPromiseThrottler.run(() => {\\ async code calling piscina.run});

[goetzrrGit]

I like the solution here with the caching.

I had a similar need recently for throttling the number of inflight promises and came up with this solution which might be inspiration for a less intrusive solution to the batching. Benefit here is that it allows us to limit the number of in process workers, BUT doesn't block on all of them completing. So we can ensure the workers are all always filled.

/** Throttle number of in flight promises. */
export class InFlightPromiseThrottler {
  private inFlightPromises: Promise<unknown>[] = [];
  private promiseLimit: number;

  public constructor(promiseLimit: number) {
    this.promiseLimit = promiseLimit;
  }

  public async run<T>(promiseFactory: () => Promise<T>): Promise<T> {
    while (this.inFlightPromises.length >= this.promiseLimit) {
      await Promise.race(this.inFlightPromises);
    }
    const promise = promiseFactory();
    this.inFlightPromises.push(promise);
    return promise.finally(() => {
      const index = this.inFlightPromises.indexOf(promise);
      if (index !== -1) {
        this.inFlightPromises.splice(index, 1);
      }
    });
  }
}

Used like

// Somewhere in a higher scope
const workerPromiseThrottler = new InFlightPromiseThrottler(8);

// When we actually do the worker run

const result = await workerPromiseThrottler.run(() => {\\ async code calling piscina.run});

Hi Dylan, Thank you for the suggestion!!! Very cool approach and I will give it a try. I hope all is well with you.

[cohansen]

Looks good to me, I saw your demo as well and it seemed like it was working well. My only question is do we need to enforce that PromiseThrottler is a singleton?

[goetzrrGit]

Looks good to me, I saw your demo as well and it seemed like it was working well. My only question is do we need to enforce that PromiseThrottler is a singleton?

I am using it as a singleton export but there is a ticket I created to use dependency injection so it wont be a singleton in the future.

#1350