Timeout during query merge

[zvictor]

While debugging a non related component of my code I wanted to temporarily reduce the load on Fauna. Thus, I added a .take(1) to my query before a .paginate call. By doing so, I got a Timeout during query merge.

/tmp/node_modules/.pnpm/fauna@0.9.0/node_modules/fauna/src/client.ts:346
        return new ServiceTimeoutError(failure, httpStatus);
               ^
ServiceTimeoutError: Read timed out (Timeout during query merge.) java.util.concurrent.TimeoutException: Timeout during query merge.
    at _Client.getServiceError_fn (/tmp/node_modules/.pnpm/fauna@0.9.0/node_modules/fauna/src/client.ts:346:16)
    at _Client.getError_fn (/tmp/node_modules/.pnpm/fauna@0.9.0/node_modules/fauna/src/client.ts:279:16)
    at _Client.query_fn (/tmp/node_modules/.pnpm/fauna@0.9.0/node_modules/fauna/src/client.ts:416:15)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async fetchMessages (file:///tmp/data-science/fetch-data.ts:15:18)
    at async processCollection (file:///tmp/data-science/fetch-data.ts:35:20)
    at async file:///tmp/data-science/fetch-data.ts:69:1 {
  httpStatus: 503,
  code: 'time_out',
  queryInfo: {
    txn_ts: undefined,
    summary: undefined,
    query_tags: undefined,
    stats: undefined
  },
  constraint_failures: undefined
}

I could reproduce the error both in fauna 0.8.0 as in 0.9.0.
Here is the broken query:

fql`Message.all().where(
    m => m.service == ${service} && m.author.username != ${ignoredAuthor}
  ).take(1).paginate(9999)`

If I change it to take(10), it works. If I remove .take, it also works.

[zvictor]

This query is also throwing the same error, but every single time:

fql`Message.where(.role)`

Same for

fql`Message.all().where(.role)`

---

.role is a key that only exist in one doc and is missing in all the other ones.
I tried different names of non existent keys and they all throw the same error.

[ptpaterson]

This is not an issue with the driver: the driver is correctly handling the 503 error returned by the database. The fact that it is timing out is an issue with the database, though.

This is possibly related to this forums topic: https://forums.fauna.com/t/fqlv10-comparison-of-objects-never-completes/3652

Where the issue is the DB hanging on a comparison between non-comparable types.

In your case, you're not providing a boolean value to .where. FQL types don't have a "truthiness" value like javascript. You need to explicitly check

fql`Message.where(.role != null)`

[ptpaterson]

Closing this, but will make a note to follow up here regarding the DB issue.

[ptpaterson]

We are still investigating the issue with .take(1).paginate(9999). But the Message.where(.role) is more likely working correctly.

The where predicate is allowed to return null and if it does so, then it is treated as false. When you run something like Message.where(.role) the expectation is that the field role is either null (missing) or a boolean value. The query starts scanning the entire Collection and resolving the role field.

.role is a key that only exist in one doc and is missing in all the other ones.

Note that the default query-timeout is 5 seconds. In this case, it is most likely that Fauna simply spent more than 5 seconds scanning, looking for a document with a role field, and found none. If it had found the one document, you would be more likely to get a 400 error with a message like error: expected type: Boolean, received String or similar.

[zvictor]

Thanks for all the info, @ptpaterson !
I have set a big query-timeout and the original error disappeared, but now I am faced with NetworkError: The network connection encountered a problem.

Config:

const client = new Client({
  query_timeout_ms: 60_000,
  http2_session_idle_ms: 60_000,
})

await client.query(fql`Message.all().where(m => m.service == ${service}).paginate(9999)`)

Error:

{
  query: {
    fql: [
      'Message.all().where(m => m.service == ',
      [Object],
      ').paginate(9999)'
    ]
  },
  arguments: undefined
}


/node_modules/.pnpm/fauna@0.9.0/node_modules/fauna/src/http-client/node-http2-client.ts:94
          throw new NetworkError(
                ^
NetworkError: The network connection encountered a problem.
    at _NodeHTTP2Client.request (/node_modules/.pnpm/fauna@0.9.0/node_modules/fauna/src/http-client/node-http2-client.ts:94:17)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async _Client.query_fn (/node_modules/.pnpm/fauna@0.9.0/node_modules/fauna/src/client.ts:385:9)
    at async fetchMessages (file:///data-science/fetch-data.ts:15:18)
    at async processService (file:///data-science/fetch-data.ts:27:18)
    at async file:///data-science/fetch-data.ts:68:1 {
  [cause]: Error [ERR_HTTP2_STREAM_ERROR]: Stream closed with error code NGHTTP2_INTERNAL_ERROR
      at new NodeError (node:internal/errors:377:5)
      at ClientHttp2Stream._destroy (node:internal/http2/core:2339:13)
      at _destroy (node:internal/streams/destroy:109:10)
      at ClientHttp2Stream.destroy (node:internal/streams/destroy:71:5)
      at Http2Stream.onStreamClose (node:internal/http2/core:549:12) {
    code: 'ERR_HTTP2_STREAM_ERROR'
  }
}

[cleve-fauna]

Reopening as @zvictor has asked another question.

[cleve-fauna]

@zvictor that idle timeout is quite aggressive. That will attempt to hold an open, but unused http2 session to Fauna for 60s. I would be surprised if Fauna is willing to do the same on the http2 session. I am speculating that that may be the cause of the NetworkError; as Fauna may simply be hanging up on you.

Can you try this without setting that timeout (and thus using the driver's default of 5s)?

There's some basis in the Node docs for this theory as well: https://nodejs.org/api/http2.html#destruction

[zvictor]

Can you try this without setting that timeout (and thus using the driver's default of 5s)?

I won't be able to work on the project I was working on for quite a while. Once I do, I can check it out, but not before that. I am sorry.

Feel free to close the issue once again.

[cleve-fauna]

Please reopen if you want more help with this.