[Bug] the concurrent modification issues in memoryPoolFactory

[neuyilan]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.8

Compute Engine

None

Minimal reproduce step

According to the current code, a table's write will have an AbstractFileStoreWrite object, which will then create a containerWriter for each bucket of each partition based on the partition. All these containerWriters share a single WriteBufferPool[1]. The code is shown below. So this may lead to the following issues:

1. The creation of WriteBufferPool is not thread safe, as the notifyNewWriter is called when writing the first data in each partition's bucket. So if multiple bucket writing threads are called simultaneously, it is not thread safe.
2. The nextSegment() function in the MemoryPoolFactory[2] retrieves the available memory from the available memory. But when multiple threads are called simultaneously, it is not thread safe either.
3. When the memory is used up and returned to the memory pool, the returnAll(List<MemorySegment> memory) function in AbstractMemorySegmentPool [3] is called. The flushWriteBuffer() function in MergeTreeWriter will trigger the returnAll function, where may cause multiple possibilities for flushWriteBuffer, such as compact or normal writing. So it is not thread safe either.

I'm not sure if this is a problem or if it was intentionally designed for it. If it's a problem, I'm willing to fix it.
Thanks.

[1] https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/operation/MemoryFileStoreWrite.java#L113
[2] https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/memory/MemoryPoolFactory.java#L140
[3] https://github.com/apache/paimon/blob/master/paimon-common/src/main/java/org/apache/paimon/memory/AbstractMemorySegmentPool.java#L59

What doesn't meet your expectations?

Fix it

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!