Mongosync Insights version 0.8.0.18

[BigMarcio]

Mongosync Insights changes include:
Mongosync log parser:

• Insights become capable of parsing mongosync_metrics.log files to visualize 40+ mongosync metrics across Collection Copy, CEA, Indexes, Verifier, and more.
• The mongosync logs plot received new metrics:
  • Source and Destination ping latency
  • Average Source CRUD event rate
  • Partitions copied
  • Total Indexes to Built and Indexes Built
  • Total Copied Data over time during the collection copy phase.
  • Estimated Oplog Time Remaining
  • Event Application Rate Per Second
  • Partitions Initialization
• Mongosync Options moved to a new tab. It now includes the /start options. And have a "Copy Markdown" feature to easily copy the options.
• New section to show common errors/warnings identified in the logs.
• Mongosync Phases now includes a table alongside the plot, making it easier to read the date and time each phase started.
• Included a tab to show which collections are/were copied in Natural Order.
• Included a tab to show Partitions Initialization.

Live Monitoring:

• Included the list of collections been copied in Natural Order.

Migration toolbox scripts were reorganized in subfolders

[Copilot]

Pull request overview

This PR updates Mongosync Insights to version 0.8.0.18, adding significant new features for parsing mongosync metrics files, displaying results in tabbed UI, detecting common errors in logs, and reorganizing toolbox scripts into subfolders.

Changes:

• Adds Prometheus metrics log parsing (mongosync_metrics.log) with a new MetricsCollector class and configurable metrics definitions via mongosync_metrics.json, plus many new log-based metrics (ping latency, CRUD rate, partitions, indexes, oplog time remaining, etc.)
• Introduces a tabbed UI for upload results (Logs, Metrics, Options, Collections/Partitions, Errors/Warnings) with filter, copy-as-markdown, and expandable error details
• Reorganizes toolbox scripts into subfolders with individual READMEs, adds new collectionDatabaseSizes script, and introduces MI_VERIFIER_CONNECTION_STRING env var and session merging

Reviewed changes

Copilot reviewed 22 out of 32 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
mongosync_plot_prometheus_metrics.py	New module for parsing Prometheus exposition format from metrics log files and generating Plotly plots
mongosync_metrics.json	Configuration for 50+ metrics across Collection Copy, CEA, Indexes, Verifier, etc.
mongosync_plot_logs.py	Extended log parsing with new metrics, file classification, metrics routing, error pattern matching, and partition init aggregation
upload_results.html	Tabbed UI with options tables, collections, partition init, errors, and markdown copy
verifier_metrics.html	Refactored to tabbed layout with improved error handling
home.html	Removed preview images, adjusted layout
file_decompressor.py	Added classified decompression yielding (line, file_type) tuples, nested archive support
app_config.py	Version bump, file classification, VERIFIER_CONNECTION_STRING, error patterns loading, TLS/certifi handling, session merge logic
mongosync_insights.py	Session merging helper, verifier connection string separation
mongosync_plot_metadata.py	Added natural order collections table, sharded resume info support, 4th row
migration_verifier.py	Removed unused functions
error_patterns.json	New config for 30+ common error patterns
probIndexesComplete/	New subfolder with index stats script and README
collectionDatabaseSizes/	New subfolder with collection size script and README
mongosyncUniqueIndexChecker/	Extended JSON coercion for index key normalization, added license
idChecker/README.md	Removed file structure section, added license
toolbox/README.md	Condensed to link to individual READMEs
VALIDATION.md, CONFIGURATION.md, README.md	Updated docs for new features and config options
templates/README.md	Deleted (content moved elsewhere)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.