Finished queries continue using memory pool on worker

[sshkvar]

Hi, after update to Trino 367 we found that finished queries doesn't release memory pool on workers.

As a result we have a lot of queries which actually finished but still exists in workers memory pool.
On the chart below you can find amount of such queries and see how these amount grown after cluster restart

As you can see we have 12000 queries which actually finished but using memory pool on workers.
Les't see example of one worker

On this worker we have 166 running queries, but 98 of them already finished but still using memory pool.
For example 20220210_093944_48617_446ev
When I try to click on this query on open in new tab I see query not found

[losipiuk]

@sshkvar thanks for the report.
Can you provide some more information about the queries you are running.

• what connectors do you use
• can you quote example problematic query?
• Is the problem always reproducable for query of a given shape or is it non-deterministic?

[sshkvar]

@losipiuk thanks for the quick reply,

These queries pretty different like simple select .... from ... or create table ... as select and in different catalogs (connectors) in our case mysql, raptor and potentially iceberg
This problem isn't always reproducible, it is randomly reproducible

[davseitsev]

Small update. All stuck queries have reservation only in ExchangeOperator:

  "queryMemoryAllocations": {
    "20220216_125714_10727_uzgkx": [
      {
        "tag": "ExchangeOperator",
        "allocation": 13218
      }
    ],
....

Amount of reserved memory is pretty small, up to a few megabytes.

[losipiuk]

@arhimondr is this manifestation of #10950? Or different thing?

[losipiuk]

And @sshkvar what Trino version did you use before 367?

[sshkvar]

And @sshkvar what Trino version did you use before 367?

We used Trino(Presto) 346

[losipiuk]

And you are not using query/task based retries which were introduced in 367?

[sshkvar]

No, we are not using retries. As I know it disabled by default

[losipiuk]

No, we are not using retries. As I know it disabled by default

Yeah.

Tricky - did you maybe have a chance to take a look into logs? Any errors/stacktraces logged?

[arhimondr]

@sshkvar Do you know if the queries you see holding memory reservations finished successfully or failed?

[sshkvar]

Tricky - did you maybe have a chance to take a look into logs? Any errors/stacktraces logged?

we didn't find any errors

@sshkvar Do you know if the queries you see holding memory reservations finished successfully or failed?

queries finished successfully

[arhimondr]

@sshkvar Do your queries contain LIMIT operation?

We've found a race condition that may result in memory not being release properly: #11088. But that shouldn't happen unless you have a LIMIT operation after exchange. For example a LIMIT after JOIN.

Also the likelihood of this problem occurrence should be fairly low. But since you are running thousands of queries it may happen that some of them trigger the race condition mentioned.

[arhimondr]

We should also improve our integration tests to verify that memory pools are empty upon completion: #11093

[losipiuk]

Should be fixed by #11088