Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OAK-10423 - Improve logging of metrics in the indexing job #1093

Merged
merged 12 commits into from Sep 8, 2023

Conversation

nfsantos
Copy link
Contributor

For each phase/task of the indexing job, log messages at the start and end following a standard format. The messages at the end include metrics in JSON format.

Example:

16:32:53.847 INFO  [mongo-dump] PipelinedMongoDownloadTask.java:131 [TASK:MONGO-DUMP:START] Starting to download from MongoDB.
...
16:32:53.866 INFO  [mongo-dump] PipelinedMongoDownloadTask.java:203 [TASK:MONGO-DUMP:END] Metrics: {"duration":"01:32:12","durationSeconds":5532,"documentsDownloaded":187187476,"enqueueingDelayMs":45,"enqueueingDelayPercentage":"0.00"}

@fabriziofortino fabriziofortino merged commit 6ec7926 into apache:trunk Sep 8, 2023
2 checks passed
mbaedke pushed a commit that referenced this pull request Sep 19, 2023
* Improve logging of metrics in the indexing job.

* Add integrity checks to the metrics formatter

* Clean up log messages

* Add logging of metrics to reindex, merge in node store and total job time.

* Improve logging

* Add missing license header.
Fix typo in class name.

* Simplify logic.

* Print elapsed time in hours:minutes instead of decimal minutes.

* Do not call .name() on reference that is potentially null.
@nfsantos nfsantos deleted the OAK-10423 branch October 25, 2023 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants