Logstash v5 Support

[theangryangel]

Logstash v5 will be out Soon(TM).

Known, possibly incomplete things to alter

• logstash-core-plugin-api dependency - Mass update plugins for 5.0.0 release elastic/logstash#4829
• Stud::Buffer replacement with multi_receive
• Exception handling - how many exceptions are really actually re-tryable?
• Tests
  • Max exception retries
  • Unretry-able exceptions
  • Connection-loss handling
  • Fatal exceptions
  • Add additional content verification (not just number of rows with correct message)
• HikariCP logging - may need to be changed in respect to however Explore use of SLF4J in Logstash and default to Log4j  elastic/logstash#4548 finishes up

[theangryangel]

Updated to api 2 in 8d27e0f

[theangryangel]

https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/exponential-threadsafe/lib/logstash/outputs/elasticsearch/common.rb appears to be the implementation that logstash prefers.

Based on that, my TODO notes:

• Remove Stud::Buffer
• Consider what exceptions we want to make retry-able and what not to (question: just how feasible is this across all drivers?)
• Do we really want it to infinitely retry? Or do we want to keep the max_exceptions ring buffer

[theangryangel]

e32b6e9 removes Stud::Buffer, keeps max_exceptions counter for a set of events, adds better support for recovering from batched insert failures. Missing tests.

[theangryangel]

Notes for myself.

Currently discovering the pain from the JDBC batch behaviour. Includes, but not limited to:

• autocommit=true+batch behaviour is undefined and driver specific
• a driver may chose to continue processing after a failure
• getting the error for a particular entry in the batch in the event of the above doesn't seem possible

Since the plugin may be used with any JDBC-compatible engine, it may be just safer for each event to be processed as a single command (like unsafe goes through), and batches go in the bin 😞

[theangryangel]

Batched SQL now in the bin.

TODO: Tests for new test cases.

[theangryangel]

Shower thought: Do we actually want to retry any non-connection related JDBC exceptions. Most seem to be fatal errors that without human intervention won't resolve by retrying.

Retryable states by class:

• 08 — Connection Exception
• 24 — Invalid Cursor State (MAYBE)
• 25 — Invalid Transaction State (MAYBE)
• 40 — Transaction Rollback (MAYBE - specifically looking at deadlocks)
• 53 — Insufficient Resources
• 54 — Program Limit Exceeded (MAYBE)
• 55 — Object Not In Prerequisite State
• 57 — Operator Intervention
• 58 — System Error

Counter thought: JDBC drivers are inconsistent, and have their own SQLSTATE codes. Also May throw an SQL exception when they dont mean to (do we even care about this?). What's worse? Holding up the pipeline, or just losing a few events.

[theangryangel]

Added custom retry_sql_states configuration. Currently this is the full sql state code, not the class.

[theangryangel]

Closing for individual issues going forward.