How safe it's to use MongoDB/NoSQL Maintenance Job

[jaro-herod]

Hello,
we are using the Polaris with MongoDB as persistence layer. Since version 1.7 there is experimental job that can be executed to perform garbage collection of orphaned entries.

I wanted to know:

• Is it safe to use it right now ( version 1.8) ?
• If no, is the plan to make it stable at 1.9 ?

[thillainirmal-tech]

The 1.8 documentation makes an important distinction:

• The MongoDB/NoSQL backend itself is still experimental (beta), so it does not have the same production-support level as the relational JDBC backend.
• Within that backend, nosql maintenance-run is the documented cleanup mechanism. The 1.8 admin-tool guide recommends running it regularly (for example, once per day), describes it as self-tuning, and says a long-running maintenance pass is safe.

So if you have already accepted the risk of using the experimental MongoDB backend, running its maintenance job is expected rather than something to avoid. I would deploy it conservatively:

1. Use the same 1.8.0 version of the admin tool and server.
2. Take a MongoDB backup/snapshot before the first production run and test against a restored copy first.
3. Run nosql maintenance-info to inspect the effective parameters.
4. Start during an off-peak window; limit object-scan-rate-limit-per-second and reference-scan-rate-limit-per-second if database load is a concern.
5. Inspect nosql maintenance-log after each early run.
6. Do not run jobs concurrently. If using the official Helm job, keep concurrencyPolicy: Forbid. Only use --supersede-run after confirming the recorded run is abandoned.

A warning that not every purgable entry was deleted is normally informational; repeated warnings mean the expected-count multiplier may need adjustment.

I would not assume a 1.9 stabilization date unless a maintainer confirms it—there is no such commitment in the published 1.8 documentation.

References:

• 1.8 metastore support levels: https://polaris.apache.org/releases/1.8.0/metastores/
• 1.8 maintenance guidance: https://polaris.apache.org/releases/1.8.0/admin-tool/#running-nosql-maintenance
• 1.8 Helm maintenance settings: https://polaris.apache.org/releases/1.8.0/helm-chart/reference/#maintenance

[dimas-b]

@jaro-herod : Polaris as a project can only provide the code "as is" without any warranty, which is noted in the license and docs.

That said, the code is stable, but bugs cannot be ruled out 100%, as I'm sure you understand 🙂

The "beta" label is just an indication that the feature is still subject to change... mainly based on user feedback.

The more people use it, the better the project quality will become. If you hit any issues, please report on the dev ML or in GH issues. PRs are also welcome if you need to adjust something to your needs.

As we see more user adoption, the "beta" label can probably be removed 😉

[dimas-b]

@jaro-herod : just to clarify: the NoSQL maintenance job is not "Garbage Collection" in the traditional Iceberg sense. It does not reclaim Object Storage space unused by tables. The Maintenance job reclaims Polaris MongoDB space occupied by catalog change history, which you no longer need. Without the Maintenance job, MongoDB size would grow indefinitely.

Cf. #5232