Better exception handling for Elastic adapter

[JSCU-CNI]

This PR fixes two issues with the Elastic adapter. Currently the Elastic adapter fails silently - as in it does not break but continues - when a record is not inserted in the index. We make sure to raise a ValueError instead so the threading.excepthook catches the error and stops the elastic writer.

The second issue we fix here is we make sure both reader and writer threads crash when self.exception is filled. This prevents a hanging reader thread if the writer thread fails.

[codecov]

Codecov Report

Attention: Patch coverage is 57.14286% with 6 lines in your changes missing coverage. Please review.

Project coverage is 82.86%. Comparing base (1f4e711) to head (a3aed88).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
flow/record/adapter/elastic.py	57.14%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #164      +/-   ##
==========================================
- Coverage   82.94%   82.86%   -0.08%     
==========================================
  Files          34       34              
  Lines        3554     3561       +7     
==========================================
+ Hits         2948     2951       +3     
- Misses        606      610       +4     

Flag	Coverage Δ
unittests	82.86% <57.14%> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yunzheng]

next to the inline comment, can you also move the following code block:

 self.exception: Exception | None = None
 threading.excepthook = self.excepthook

to before self.es = elasticsearch.Elasticsearch().

This ensures that if it fails during instantiating ElasticSearch(), that self.exception is available and does not cause another exception during close().

[JSCU-CNI]

We added a simple pure python progress bar for the elastic writer in 06cfa15. tqdm is a minimal and fast dependency and allows one to see how many records have been processed by rdump in a TTY.

$ target-query -q -f mft example.img | rdump -w "elastic+http://foo:bar@localhost:9200?verify_certs=0&index=test"
[reading from stdin]
2199501 records [06:19, 5041.39 records/s]

[Schamper]

We added a simple pure python progress bar for the elastic writer in 06cfa15. tqdm is a minimal and fast dependency and allows one to see how many records have been processed by rdump in a TTY.

$ target-query -q -f mft example.img | rdump -w "elastic+http://foo:bar@localhost:9200?verify_certs=0&index=test"

[reading from stdin]

2199501 records [06:19, 5041.39 records/s]

On the topic of progress bars, I thought of a mechanism recently that will generically allow any dissect.target plugin to provide progress "feedback", making it possible to provide actual completion progress bars for plugins where that info is calculatable, and more generic progress bars for plugins where it isn't.

I plan to write down these concepts in a few issues next week.

[yunzheng]

We added a simple pure python progress bar for the elastic writer in 06cfa15. tqdm is a minimal and fast dependency and allows one to see how many records have been processed by rdump in a TTY.

$ target-query -q -f mft example.img | rdump -w "elastic+http://foo:bar@localhost:9200?verify_certs=0&index=test"

[reading from stdin]

2199501 records [06:19, 5041.39 records/s]

On the topic of progress bars, I thought of a mechanism recently that will generically allow any dissect.target plugin to provide progress "feedback", making it possible to provide actual completion progress bars for plugins where that info is calculatable, and more generic progress bars for plugins where it isn't.

I plan to write down these concepts in a few issues next week.

I ❤️ progress bars.

To comment on this from the flow.record perspective, I've always wanted to add progress bar support and I think it's best if we split the tqdm/progress bar support in the elastic adapter into a separate issue and add this to tools/rdump.py so all adapters can benefit from it.

[yunzheng]

It would be great if the tqdm code can be moved and incorporated in a new PR. See #165

[JSCU-CNI]

It would be great if the tqdm code can be moved and incorporated in a new PR. See #165

Created a separate PR in #166.