Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add logging for bulk insert and quiet mode to avoid printing tracebacks #14

Closed
wants to merge 1 commit into from

Conversation

linkdd
Copy link
Contributor

@linkdd linkdd commented Jul 7, 2017

There was no exception catching for bulk insert operations, which would end up with mongo-connector crashing and stopping.

With this feature, I'm adding logging of erronous document's primary key and namespace.

I also added an option quiet (default: False) which, when enable, avoid printing the exception traceback (the feature in pull request #13 can create big requests which is noise in the log file).

@coveralls
Copy link

coveralls commented Jul 7, 2017

Coverage Status

Coverage decreased (-2.7%) to 93.587% when pulling 8257878 on linkdd:feature-log-errors into cacc0f6 on Hopwork:master.

@MaximeGaudin
Copy link

Hi @linkdd !

I will not merge this contribution for the following reasons :

  • Crashing is easily monitorable (that's what we are doing as a matter of fact) whereas scanning log requires I/O for reading and CPU for pattern matching. Also, it introduces a dependency between the software (via the format of the log) and the monitoring tools (via the pattern to find). A viable solution to this is implementing a health check though
  • Crashing was a clear behavior and "guaranteed" to left the base in an incomplete but consistent state. By not crashing, the bulk insert is aborted but the real-time operations are steal streamed to the base. I might be wrong but it seems that you can find yourself with a partial bulk, a gap and the new data.

What do you think ?

@linkdd linkdd closed this Dec 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants