Elastic adapter #2727

camgunz · 2024-04-22T16:00:18Z

Description:

This PR is to add an elasticsearch source.

Checklist:

[*] Tests passing (make test-community)?
- Looks like some issues w/ the tar source?
[*] Lint passing (make lint this requires golangci-lint)?
- unrelated issues in LDAP seems like

CLAassistant · 2024-04-22T16:00:24Z

All committers have signed the CLA.

- Added tests - Added some documentation comments - Threaded the passed context through to all the API requests

zricethezav · 2024-05-23T16:09:01Z

pkg/sources/elasticsearch/elasticsearch.go

+		unitsOfWork := distributeDocumentScans(&indices, s.concurrency, scanCoverageRate)
+
+		for uowIndex, outerUOW := range unitsOfWork {
+			uow := outerUOW


@camgunz
Suggestion: Even though uowIndex is only used in logs, it still needs to have a copy similar to uow inside the loop otherwise we have the potential for incorrect log output

Oof, nice catch; I think this started life as _ but then I used it for logging and missed pulling it in 👍

pkg/sources/elasticsearch/elasticsearch.go

zricethezav · 2024-05-23T16:22:34Z

pkg/sources/elasticsearch/unit_of_work.go

+	}
+}
+
+func (uow *UnitOfWork) AddSearch(


Question: any reason we need to export this function? Looks like the only place it's being used is distributeDocumentScans and the tests.

Yeah you're right, no need to export here 👍

pkg/sources/elasticsearch/api.go

zricethezav · 2024-05-23T17:11:51Z

pkg/sources/elasticsearch/api.go

+
+func (i *Index) DocumentAlreadySeen(document *Document) bool {
+	i.lock.Lock()
+	defer i.lock.Unlock()


Suggestion: rather than locking at the top and deferring the unlock we should just surround contentious access with straight up lock/unlocks rather than defer. Easier to figure out what needs to be protected that way.

Interesting, yeah I buy that. I'll change what I can -- in this case I'll probably keep defer and just move it below ParseTimestamp because everything else needs to be write synchronized, but I think the others can all be explicit.

pkg/sources/elasticsearch/api.go

zricethezav · 2024-05-23T17:19:14Z

pkg/sources/elasticsearch/api.go

+		return "", err
+	}
+
+	pitID, found := data["id"].(string)


nit: use ok instead of found for consistency

pkg/sources/elasticsearch/api.go

pkg/sources/elasticsearch/elasticsearch.go

… in subsequent scans

…stamp range clause in searches

…ronized part of the code clearer

camgunz requested review from a team as code owners April 22, 2024 16:00

Charlie Gunyon and others added 17 commits May 10, 2024 16:25

Add stub source and elastic API funcs

998c893

Spawn workers and ship chunks

b9a7c9a

Now successfully detects a credential

e6ea20d

- Added tests - Added some documentation comments - Threaded the passed context through to all the API requests

Linting fixes

4624cef

Add integration tests and resolve some bugs they uncovered

c6bb4b8

Logstash -> Elasticsearch

ca89bc2

Add support for --index-pattern

8b1dda6

Add support for --query-json

dc4397e

Use structs instead of string building to construct a search body

b7bd504

Support --since-timestamp

35748c9

Implement additional authentication methods

ae2cfe5

Fix some small bugs

a8dda1d

Refactoring to support --best-effort-scan

f862ac4

Finish implementation of --best-effort-scan

ef99cf9

Implement scan catch-up

96f2c44

Finish connecting support for nodes CLI arg

7564385

Add some integration tests around the catchup mechanism

ced2a76

camgunz force-pushed the elastic-adapter branch from 49cb40f to ced2a76 Compare May 10, 2024 15:01

camgunz and others added 6 commits May 10, 2024 17:02

Merge branch 'main' into elastic-adapter

131d0a7

go mod tidy

0eda2ac

Fix some linting issues

826fc8b

Merge remote-tracking branch 'upstream/main' into elastic-adapter

d566f2a

Remove some debugging Prints

4b3e259

Move off of _doc

27b274e