optimize: add a descendant map to speed up v5 export performance

[ZxBing0066]

Background

INS-694

When there are more than 1000 requests in one collection, some git operations will significantly block the main process, and users can't do anything during that time. This has seriously affected the user experience.

Changes

The performance of withDescendants is a bit worse because it always queries all types for each item, so I added a queryTypesDescendantMap to avoid unnecessary queries.

• Support queryTypesDescendantMap in db.withDescendants.
• Use WORKSPACE_EXPORT_TYPES_DESCENDANT_MAP to declare the export types.

[Copilot]

Pull Request Overview

This PR introduces a descendant‐type map to optimize withDescendants lookups during v5 exports, reducing unnecessary queries and improving export performance.

• Support passing a queryTypesDescendantMap to database.withDescendants
• Declare the export‐type descendant relationships in WORKSPACE_EXPORT_TYPES_DESCENDANT_MAP
• Update export logic and tests to leverage the new map

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
packages/insomnia/src/models/index.ts	Added WORKSPACE_EXPORT_TYPES_DESCENDANT_MAP constant
packages/insomnia/src/common/insomnia-v5.ts	Passed the descendant map into withDescendants in the export flow
packages/insomnia/src/common/database.ts	Extended withDescendants to accept and apply a descendant‐type map
packages/insomnia/src/common/tests/database.test.ts	Added tests covering the new descendant‐map behavior

Comments suppressed due to low confidence (3)

packages/insomnia/src/common/database.ts:621

• Consider adding a JSDoc comment for queryTypesDescendantMap to explain its purpose, expected key format, and fallback behavior when a type is missing from the map.

queryTypesDescendantMap?: Record<string, string[]>,

packages/insomnia/src/common/insomnia-v5.ts:457

• The workspaceDescendants variable is assigned but never used—either use it to filter export items or remove this assignment to avoid dead code.

const workspaceDescendants = await database.withDescendants(

packages/insomnia/src/common/database.ts:641

• The order of types determines traversal order and can lead to nondeterministic result ordering; consider sorting or explicitly defining type order to ensure consistent descendant lists.

const types = queryTypesFromDescendantMap ?? (queryTypes?.length ? queryTypes : allTypes());

[ZxBing0066]

Ideally, we should declare all the above exported types with the descendant types, but I cannot find some of them, like proto file and profo dir.
Not sure if there are any nesting relations I missed.

[ihexxa]

It may be related to grpc but i didnt find them either.

[ZxBing0066]

It always goes through all types to find out all the valuable items. But in the real world, it's unnecessary. E.g., there is no CookieJar that uses Request as its parent.

[ZxBing0066]

Comparison with the following code:

    console.time('[debug] database.withDescendants');
    await database.withDescendants(workspace);
    console.timeEnd('[debug] database.withDescendants');

    console.time('[debug] database.withDescendants with queryTypes');
    await database.withDescendants(workspace, null, exportableTypes);
    console.timeEnd('[debug] database.withDescendants with queryTypes');

    console.time('[debug] get-workspace-from-loader');
    await getWorkspaceWithDescendants({ workspaceId });
    console.timeEnd('[debug] get-workspace-from-loader');

    console.time('[debug] database.withDescendants with descendantMap');
    const workspaceDescendants = await database.withDescendants(
      workspace,
      null,
      [],
      WORKSPACE_EXPORT_TYPES_DESCENDANT_MAP,
    );
    console.timeEnd('[debug] database.withDescendants with descendantMap');

Almost 100x for a workspace with 100 requests.

16:46:55.044 › [debug] database.withDescendants: 337.384ms
16:46:55.255 › [debug] database.withDescendants with queryTypes: 210.817ms
16:46:55.262 › [debug] get-workspace-from-loader: 6.435ms
16:46:55.265 › [debug] database.withDescendants with descendantMap: 3.51ms
16:46:55.606 › [debug] database.withDescendants: 322.907ms
16:46:55.814 › [debug] database.withDescendants with queryTypes: 207.675ms
16:46:55.821 › [debug] get-workspace-from-loader: 6.911ms
16:46:55.825 › [debug] database.withDescendants with descendantMap: 3.448ms
16:46:56.169 › [debug] database.withDescendants: 326.421ms
16:46:56.373 › [debug] database.withDescendants with queryTypes: 203.712ms
16:46:56.380 › [debug] get-workspace-from-loader: 6.297ms
16:46:56.384 › [debug] database.withDescendants with descendantMap: 4.107ms
16:46:56.418 › [debug] database.withDescendants: 17.164ms
16:46:56.427 › [debug] database.withDescendants with queryTypes: 9.568ms
16:46:56.432 › [debug] get-workspace-from-loader: 4.014ms
16:46:56.434 › [debug] database.withDescendants with descendantMap: 2.051ms

[ihexxa]

as queryTypes is an existing arg and probably it should be intersected with the result from queryTypesDescendantMap

[ihexxa]

It may be related to grpc but i didnt find them either.

[ZxBing0066]

Hi @gatzjames, I made some changes to withDescendants, here's the performance test data:

[main] 15:52:44.797 › withDescendantsV0 A rich collection: 99.353ms
[main] 15:52:44.799 › withDescendantsV1 A rich collection: 2.176ms
[main] 15:52:44.803 › withDescendantsV2 A rich collection: 3.166ms
[main] 15:52:44.806 › withDescendantsV3 A rich collection: 3.23ms
[main] 15:52:44.931 › withDescendantsV0 A Design Document: 114.453ms
[main] 15:52:44.934 › withDescendantsV1 A Design Document: 2.237ms
[main] 15:52:44.938 › withDescendantsV2 A Design Document: 4.254ms
[main] 15:52:44.941 › withDescendantsV3 A Design Document: 3.225ms
[main] 15:52:46.515 › withDescendantsV0 400 folders: 1.557s
[main] 15:52:46.566 › withDescendantsV1 400 folders: 50.712ms
[main] 15:52:46.584 › withDescendantsV2 400 folders: 17.949ms
[main] 15:52:46.593 › withDescendantsV3 400 folders: 8.387ms
[main] 15:52:46.759 › withDescendantsV0 20 requests: 54.689ms
[main] 15:52:46.761 › withDescendantsV1 20 requests: 1.552ms
[main] 15:52:46.764 › withDescendantsV2 20 requests: 2.66ms
[main] 15:52:46.766 › withDescendantsV3 20 requests: 2.129ms
[main] 15:52:47.191 › withDescendantsV0 200 requests: 419.861ms
[main] 15:52:47.194 › withDescendantsV1 200 requests: 2.631ms
[main] 15:52:47.202 › withDescendantsV2 200 requests: 7.839ms
[main] 15:52:47.205 › withDescendantsV3 200 requests: 3.413ms
[main] 16:05:07.402 › withDescendantsV0 80 folders: 320.614ms
[main] 16:05:07.414 › withDescendantsV1 80 folders: 11.661ms
[main] 16:05:07.419 › withDescendantsV2 80 folders: 4.866ms
[main] 16:05:07.422 › withDescendantsV3 80 folders: 2.961ms

withDescendantsV0(develop version) -> withDescendantsV1(first version in this PR) -> withDescendantsV2(the version after your change) -> withDescendantsV3(latest version).

It's slower when dealing with a workspace in most cases, but faster when dealing with cases where there are many parent items at the same level.