Feasibility of in-place/partial updates for large documents

[calebj]

On the Discord call earlier today, I asked a question about how large updates were handled, and if there were any plans to match the performance of MongoDB on this front. A discussion makes more sense than an issue.

I haven't worked on the internals of Postgres, MongoDB or WiredTiger, but my applications perform lots of small updates in large documents. Without "normalizing" the documents so full updates can be smaller and faster, it will be harder for DocumentDB to compete in this use case.

MongoDB uses this heuristic when deciding whether to rewrite a record in the WT datastore during an update:

• The existing record is larger than 1KiB,
• The change can be made with 16 or fewer WT_MODIFY operations, changing no greater than 10% of the data; and
• The new version is no more than 10% larger than the old one

The issues and discussion on the Mongo side when this was implemented are WT-2792 and SERVER-29250. Since WT handles in-place updates with MVCC, matching that almost certainly requires implementing a new access method.

[patty-chow]

Thanks for the question. DocumentDB doesn't support in-place updates like MongoDB’s WT_MODIFY. While we target the document by TID, PostgreSQL creates a new row and marks the old one dead. Because most documents have secondary indexes, we can’t take advantage of HOT updates, so even small changes result in full rewrites.

There’s ongoing work in the PG community to do just that, and we'd be more than happy to contribute in any way we can. In the meantime, we’re exploring ways to reduce overhead for update-heavy workloads. If you’re up for sharing more details on your document size and update patterns, that’d be super helpful.