ArangoDB 3.10.0 "gauge arangodb_search_num_failed_commits already exists"

[Sieabah]

Attempting to register the following index creates the following error. This is hitting server version 3.10.0 on docker. No logs on the server relating to this guage.

  {
    name: 'verification-attempt-identities',
    type: 'inverted',
    fields: ['attemptIdentities[*].value'],
  },

ArangoError: gauge arangodb_search_num_failed_commits already exists (exception location: /work/ArangoDB/arangod/Metrics/MetricsFeature.cpp:83). Please report this error to arangodb.com
2022-10-20T15:48:42.822273765Z     at new ArangoError (/srv/app/node_modules/src/error.ts:142:17)
2022-10-20T15:48:42.822277435Z     at callback (/srv/app/node_modules/src/connection.ts:671:19)
2022-10-20T15:48:42.822279905Z     at IncomingMessage.<anonymous> (/srv/app/node_modules/src/lib/request.node.ts:172:15)
2022-10-20T15:48:42.822281765Z     at IncomingMessage.emit (node:events:525:35)
2022-10-20T15:48:42.822283895Z     at IncomingMessage.emit (node:domain:489:12)
2022-10-20T15:48:42.822285395Z     at endReadableNT (node:internal/streams/readable:1358:12)
2022-10-20T15:48:42.822287005Z     at processTicksAndRejections (node:internal/process/task_queues:83:21)
2022-10-20T15:48:42.822580284Z 10/20/2022, 3:48:42 PM [ExceptionHandler] gauge arangodb_search_num_failed_commits already exists (exception location: /work/ArangoDB/arangod/Metrics/MetricsFeature.cpp:83). Please report this error to arangodb.com +0ms
2022-10-20T15:48:42.822600244Z ArangoError: gauge arangodb_search_num_failed_commits already exists (exception location: /work/ArangoDB/arangod/Metrics/MetricsFeature.cpp:83). Please report this error to arangodb.com
2022-10-20T15:48:42.822604434Z     at new ArangoError (/srv/app/node_modules/src/error.ts:142:17)
2022-10-20T15:48:42.822606384Z     at callback (/srv/app/node_modules/src/connection.ts:671:19)
2022-10-20T15:48:42.822608214Z     at IncomingMessage.<anonymous> (/srv/app/node_modules/src/lib/request.node.ts:172:15)
2022-10-20T15:48:42.822610354Z     at IncomingMessage.emit (node:events:525:35)
2022-10-20T15:48:42.822612004Z     at IncomingMessage.emit (node:domain:489:12)
2022-10-20T15:48:42.822613594Z     at endReadableNT (node:internal/streams/readable:1358:12)
2022-10-20T15:48:42.822614994Z     at processTicksAndRejections (node:internal/process/task_queues:83:21)

[pluma4345]

Hi @Sieabah, thanks for reporting this issue. I can't see anything arangojs-specific in what's causing this error so I've transferred the issue to the main ArangoDB repo.

[dothebart]

do you have similar named indices on other databases or collections?

[MBkkt]

Hi,
please note that index names should be unique per collection.
The error message which you got unclear but it indicate about that.
If you don't have same index name, probably it's a bug that was fixed in devel (will be in 3.10.1).
So I'm closing this question as answered.
Cheers

[Sieabah]

Sorry for the delay, I ended up with covid and lost track of this. @MBkkt I'm using the "ensure index" call which just attempts to register if it doesn't already exist. This works with persistent indexes just fine it's only inverted indexes that cause this prometheus gauge issue.

async ensureIndexExists(arangoModel: ArangoModel<any>) {
    const { ensureIndexes = true } = readArangoMetadata(arangoModel.constructor);
    if (!ensureIndexes) {
      return;
    }

    const indexName = (name) => `${this.collection.name}_${name}`;

    const desiredIndexes = new Set(
      arangoModel.indexConfig.map(({ name }) => indexName(name))
    );
    desiredIndexes.add("primary");

    const indexesToRemove =
      ensureIndexes === "create-only"
        ? []
        : await arangoModel.collection
          .indexes()
          .then((indexes) =>
            indexes
              .filter((index) => !desiredIndexes.has(index.name || index.id))
              .map(({ id }) => id)
          );

    try {
      await Promise.all([
        ...indexesToRemove.map((id) => {
          this.logger.log(`Destroying index ${id}`);
          return arangoModel.collection.dropIndex({ id })
        }),
        ...arangoModel.indexConfig.map(({ ...index }: any) =>{
            this.logger.log(`Ensuring index ${index.name}`);
            return arangoModel.collection.ensureIndex(
              Object.assign(index, {
                name: indexName(index.name),
                inBackground: index.inBackground ?? true,
              })
            );
          }
        ),
      ]);
    } catch(err) {
      this.logger.error(`Error while ensuring indexes for ${arangoModel.collection.name}`);
      this.debug(err);
      throw err;
    }
  }

The array this is pulling from is just defining indexes like so, which are all uniquely named. All persistent indexes worked fine, but inverted indexes cause this gauge issue. Is this something unique to inverted indexes where calling ensureIndex() on an existing inverted index causes a prometheus gauge to be double defined?

      {
        name: 'user-apex-ids',
        type: 'inverted',
        fields: ['apexIds'],
      },
      {
        name: 'user-auth-id',
        type: 'persistent',
        fields: ['authId'],
        unique: false,
      },

The index in question is defined and is returned

{
   analyzer: 'identity',
   cleanupIntervalStep: 2,
   commitIntervalMsec: 1000,
   consolidationIntervalMsec: 1000,
   consolidationPolicy: {
     type: 'tier',
     segmentsBytesFloor: 2097152,
     segmentsBytesMax: 5368709120,
     segmentsMax: 10,
     segmentsMin: 1,
     minScore: 0
   },
   features: [ 'frequency', 'norm' ],
   fields: [ [Object] ],
   id: 'auth_nau_users/4722102',
   includeAllFields: false,
   name: 'collection_user-apex-ids',
   primarySort: { fields: [], compression: 'lz4' },
   searchField: false,
   sparse: true,
   storedValues: [],
   trackListPositions: false,
   type: 'inverted',
   unique: false,
   version: 1,
   writebufferActive: 0,
   writebufferIdle: 64,
   writebufferSizeMax: 33554432
}

The persistent index returns like normal, however calling ensure index with the same name and unchanged configuration returns without issue.

{
   cacheEnabled: false,
   deduplicate: true,
   estimates: true,
   fields: [ 'authId' ],
   id: 'auth_nau_users/4721604',
   name: 'collection_user-auth-id',
   selectivityEstimate: 0.5263157894736842,
   sparse: false,
   type: 'persistent',
   unique: false
},

Regardless I don't think this error is correctly reporting. It certainly isn't consistent with other index types when it comes to ensureIndex

[MBkkt]

This works with persistent indexes just fine it's only inverted indexes that cause this prometheus gauge issue.

Because only inverted have metrics per index now, if some other index will have it, it will be same error
But in general this problem not about metrics, it about implementation of index creation, for non inverted index ensureIndex relies on current implementation (there is first init step indexes is stub (not true for inverted))

Anyway you right and it should be fixed

[Simran-B]

Internal ticket that tracks this issue: https://arangodb.atlassian.net/browse/SEARCH-385

[MBkkt]

Fixed in nightly https://download.arangodb.com/nightly/3.10/index.html

Also will be available in 3.10.2