Editor piece tree chunk management

[bpasero]

I am playing around with reading and writing very large files (27 MB) when sandbox is enabled where the file service is implemented via remote IPC (same as when you are connected to a remote such as SSH).

For some unrelated reason, our call into IFileService.readFileStream ends up with a small chunk of 256kb and a large chunk of the rest instead of N chunks of 256kb each (which is what we end up getting when sandbox is not enabled). We end up passing this into the PieceTreeTextBufferBuilder#acceptChunk method.

Later, when I trigger save, we use the snapshot method to have a Readable we can iterate over to write to disk. I end up with the exact 2 chunks as before. So we end up with a small write and a very large write instead of smaller chunks.

Is this the intended behaviour or does the piece tree have some own logic for managing chunks? Is it potentially dangerous to have such a large chunk or is that fine?

Steps to reproduce:

• produce a large text file with 1m lines
• run with scripts/code.sh --__sandbox
• open the file and have a breakpoint where we create the piece tree

[bpasero]

Here is the file that I have used for reproduction:
Features.txt.zip

[bpasero]

I did some more testing and was able to figure out why reading the large file produces only 2 chunks and I noticed that this only happens for the very first file we open because we load the iconv-lite encoding library lazily and while the library loads, the chunks get buffered.

As soon as I test this subsequently, situation is a bit better and many chunks are used in the piece tree.

So far, my strategy in all of the VSBuffer handling code I wrote was to VSBuffer.concat all the chunks once a consumer is there. This has the benefit that a consumer will end up with a single chunk instead of many, but evidently this also means the piece tree is sparse.

[alexdima]

From my understanding of the piece tree text buffer, having a huge chunk in there should not cause any problems. @rebornix what do you think ?

[rebornix]

@alexdima the text buffer will keep the buffers/chunks untouched so it doesn't matter if the buffer is a huge chunk, as long as VSBuffer is not doing string concatenation (as PieceTree will do charAt reads which might flatten concatString)

[alexdima]

👍 @bpasero we should be fine even with this anomaly.