2024 Ingestion Updates #1

nathanorwick · 2024-06-07T16:53:08Z

Cluster mode now as a default for Druid
Updates for the new Druid extension interface API
New Mongo collection and URI
New Redis URI
Fixed null issue for prefix parameter
Update test with new GISJOINs (and now passing all tests with the Redis cluster)
New CustomTextOutputFormat class for Map Reduce because the new Hadoop version wasn't playing nicely with removing tabs

We may want to make host and port optional Parameters for create instead of hard coding them still (as lattice-132 this time around).

nathanorwick · 2024-06-07T20:00:51Z

Tests are only breaking in github, not on the target environment.

sustain-project and others added 6 commits May 20, 2024 12:27

hopefully this easy

a03bf62

update mongo uri & test for new tract data

6d1ab8b

remove tab in madreduce output and SOP in create

a85f838

updatedruid version and make cluster mode default

9e3ece6

change redis host and port

42dcb55

ensure tests pass for cluster setup as default

8951d57

nathanorwick assigned paige-marie Jun 7, 2024

paige-marie added 5 commits June 7, 2024 11:16

fix tests breaking on github action

498adac

conditional factories test

6bf9ae2

final conditional test?

03c8766

rework conditional test logic

75e8281

another non-cluster issue

ed2727c

geosieve-mapper optional tests

7e003dc

Provide feedback