Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SOLR-14443: Make SolrLogPostTool resilient to odd requests #1525

Conversation

gerlowskija
Copy link
Contributor

@gerlowskija gerlowskija commented May 18, 2020

Description

SolrLogPostTool is relatively brittle in how it indexes log records to Solr. A single error from Solr causes the tool to crash. Further, when user requests have duplicate or unnecessary parameters, the tool attempts to record all of these (even when the field used is singlevalued).

As a result, a single "oddball" request crashes the whole ingestion run.

Solution

This PR makes two related changes. First, it changes SolrLogPostTool's batching code to log (but ultimately ignore) errors. This allows an indexing run to survive malformed records or other indexing blips. Second, it changes the record-parsing code to only record single values for single-valued log record fields. This prevents a common cause of malformed records.

Tests

Manual testing, in addition to a test case added to SolrLogPostToolTest

Checklist

Please review the following and check all that apply:

  • I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
  • I have created a Jira issue and added the issue ID to my pull request title.
  • I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
  • I have developed this patch against the master branch.
  • I have run ant precommit and the appropriate test suite.
  • I have added tests for my changes.
  • I have added documentation for the Ref Guide (for Solr changes only).

@gerlowskija gerlowskija merged commit 78f4a5b into apache:master May 22, 2020
@gerlowskija gerlowskija deleted the SOLR_14443_make_log_post_tool_more_resilient_to_odd_requests branch May 22, 2020 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant