Sometimes ensureIndexes fails

[zachlankton]

UPDATE (TL;DR)

For those searching this issue and looking for a quick fix: a forked and patched version of ottoman is available at https://github.com/zachlankton/node-ottoman/tree/610-ensure-fix.
This will only be maintained until the patch is published in the official repo. ( Currently pending PR: #611 )

It can also be installed with NPM:

npm install ottoman@npm:@zachwritescode/ottoman

Alternatively, you can update your package.json dependencies with:

"ottoman": "npm:@zachwritescode/ottoman@^2.0.1",

Original Post:

Hello, I am experiencing an intermittent error.

Ensuring Indexes...
[Error: LCB_ERR_KEYSPACE_NOT_FOUND (404): Keyspace is not found (collection or bucket does not exist)] {
  code: 404,
  ctxtype: 'query',
  first_error_code: 12003,
  first_error_message: 'Keyspace not found in CB datastore: default:connext._default.UserProfile',
  statement: 'BUILD INDEX ON `connext`.`_default`.`UserProfile`(`Ottoman_defaultUserProfile`) USING GSI',
  client_context_id: 'd9c1ccc274582b8e',
  parameters: '',
  http_response_code: 500,
  http_response_body: ''
}

This only happens sometimes when calling the await start() stand alone function. However, this error never occurs when I do the following:

await ottoman.connect(cnx)
await ottoman.ensureCollections()
await pause(1000); //Helper function to force delays
await ottoman.ensureIndexes();

I've written a test script that reproduces the problem pretty reliably ( strange for an intermittent problem 😊 )
It also runs a separate set of tests using the above solution as well as using the couchbase node.js SDK directly with no pause and also with a pause between creating the collection and creating the index.

It appears the main issue is needing a delay between creating collection and creating indexes, regardless if done through ottoman or couchbase node sdk?

You can get the test script here https://gist.github.com/zachlankton/aaaa7d6d1cc1ee34e0974bc1ee05b191

Here is a snapshot of the results I'm getting from this script on my machine:

PS C:\Users\zachl\Documents\ottoman-test> node ottoman-test.mjs

Testing Ottoman with Start....
Had 5 errors out of 10 tests 

Testing Ottoman with Pause....
Had 0 errors out of 10 tests 

Testing with Couchbase....
Had 7 errors out of 10 tests 

Testing with Couchbase Pause...
Had 0 errors out of 10 tests 

[zachlankton]

I am also using the latest couchbase docker image with these tests on my Windows Machine:

[System Summary]

Item	Value	
OS Name	Microsoft Windows 10 Home	
Version	10.0.19042 Build 19042	
System Manufacturer	Eluktronics Inc.	
System Model	MAX-15	
System Type	x64-based PC	
System SKU	MAX-15 G1	
Processor	Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz, 2304 Mhz, 8 Core(s), 16 Logical Processor(s)
Installed Physical Memory (RAM)	64.0 GB		

[AV25242]

Hello @zachlankton thanks for reaching out.
Creation of collections and indexes takes time and how long depends on your use-case, the general recommendation is for you to create these entities offline but we do realize that you would need test coverage just like in this case so what you are seeing is an intended behavior.

Now having said that we introduced a configuration called retryCount. By default the count is 3 you can scale it to suit your needs and see if that helps solve the problem. Basically it encapsulates the delay that you manually introduced in your code.

Coming to the Indexes, its a similar behavior, just that you are not seeing it because you are not using Indexes immediately, had you been running a query that uses the Indexes you would have noticed a failure if the Indexes are not created.

To overcome Indexes concerns you can turnoff creation of Indexes by passing in ignoreWatchIndexes: true and register to the indexReadyHook which will basically notify you when Indexes are done building.

Once again we strongly recommend creation of scopes/collections/indexes offline in production environments in away that calling ottoman.start doesn't create anything and the process is quick.

Hope this helps !

[zachlankton]

Hello @AV25242 Thanks for the response!

I completely understand that for production we would have the db and indexes already setup and we would not use start() or ensureCollections() or ensureIndexes().

But for testing purposes and development this is exactly what I want! The documentation leads me to believe that calling start() will guarantee that my collections and my indexes will be ready so that I can run queries on them without errors. Here it seems the index creation is failing because the collection does not exist (or is not ready) yet?

If this is truly intended behavior then the documentation is not very clear here. As it stands right now ensureCollections() works about 50% of the time for me. The only sure fire way to make sure my collection is ready before running ensureIndexes() is to force a delay.

I notice the ensureIndexes() does not have the same retry logic as ensureCollections() that you mentioned... is this possibly what needs to be implemented in ensureIndexes()?

If this seems reasonable, I can work on a fix to contribute this solution.
Let me know what you think.

[AV25242]

hello @zachlankton I agree with you that there is a lot of scope to improve the documentation in this regard.
Yes the index creation is failing because the collection didn't exist in your example (it was in the process of being created).
ensureIndexes() helps build indexes which is more time consuming than creation of scopes and collections and that would be one of the reasons why we dont provide you with the option of retry but instead we provide you a callback with the help of hooks like mentioned before, the hook will let you know when the Index creation is completed.

With respect to contribution, we would love to see contributions from you in any space i.e documentation or code PR.

Also if may I take the liberty to ask, what is that you are building using Ottoman, curious to learn your usecase.

[zachlankton]

Thanks @AV25242 ... I believe there was some misunderstanding here. The error LCB_ERR_KEYSPACE_NOT_FOUND was not handled in a way that would provide for what you describe. The error persisted even with the ignoreWatchIndexes set to true and hooking into the callback. This is because the collection was not ready, so the index could not be built.

I cloned the repo and ran the test suite on the couchbase/server-sandbox database image and all tests pass wonderfully in legacy tests! I see that the non legacy tests fail for this same error that I am reporting (as well as query Planning Failures most likely due to the index not being ready)

I modified the tryCreateCollection logic to guarantee the collection is ready and fixed the first 6 failing tests for the LCB_ERR_KEYSPACE_NOT_FOUND.

I reviewed the code and found that the "if" block for ignoreWatchIndexes will ONLY run if the indexOnlinePromise is defined... and indexOnlinePromise would only get defined in ensure_n1ql_indexes.ts IF there were indexReadyHooks registered. I refactored the logic so that ensureIndexes will return the indexOnlinePromise as it should.

Now all Legacy and non Legacy Tests pass perfectly!

I will have a pull request prepared shortly.

[zachlankton]

@AV25242 To answer your other question... I am actually just learning about couchbase and ottoman currently. I have used it in some small projects for the company that I work for, nothing heavy, but I really love the developer experience that I have had so far with couchbase and ottoman, its awesome!!

The way I stumbled on this issue was when trying to build a database connector for next-auth using ottoman. So I was trying to leverage the start() command to ensure the collections and indexes were ready before running my queries.... That's when I started noticing that sometimes the queries would fail in my testing... and one project begets another... here I am! 😂

[ejscribner]

Hey @zachlankton! Wanted to update you that this has been added in the latest release of Ottoman (v2.1.0), available now on NPM. Thanks for the support!

[zachlankton]

Awesome! Thank you!

[AV25242]

Closing this issue, hoping its resolved @zachlankton if not please open a follow-up ticket