Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fields with a state of DROP on normalization failure #2267

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
6f55333
Fields with a state of DROP on normalization failure are not written …
rdhayes68 Feb 7, 2024
2911976
Create docker images in CI (#2264)
alerman Feb 7, 2024
0fd8475
updated common-utils submodule and updated quickstart docker to not c…
jwomeara Feb 8, 2024
93ddde2
Do not reduce fields when preloading iterator options
apmoriarty Feb 8, 2024
4c51cb6
Enforce term limits during EdgeQueries (#2274)
jschmidt10 Feb 8, 2024
3e80d99
Change definedView to be a map definedViews (#2227)
alerman Feb 9, 2024
262cdde
Added remote edge query logic and tests (#2250)
ivakegg Feb 12, 2024
35561c1
Updates to try and avoid hanging ivarators (#2093)
ivakegg Feb 12, 2024
7afbc59
Clearing noisy logs in start-loader (#2258)
avgAGB Feb 12, 2024
8a05595
1799 add codestyle (#1811)
cogross Feb 12, 2024
13f196e
Use the pluginEnvironment instead of AccumuloConfiguration as the (#2…
ivakegg Feb 12, 2024
cf9d8cf
Enrich 'i' and 'ri' rows in metadata table with event date (#2094)
lbschanno Feb 13, 2024
efb4e28
6.7.0-SNAPSHOT
ivakegg Feb 13, 2024
05a4a12
#2226 - fix javadoc warnings (#2232)
cogross Feb 14, 2024
4a92fa8
puller version into sync with the rest of datawave
ivakegg Feb 14, 2024
6fbd222
2170 - update dependencies (#2176)
cogross Feb 14, 2024
2f370d3
2013 - jexl uuid query parser research/changes
Nov 2, 2023
2a269b3
Improved SSDeep ingest and query test framework (#2230)
drewfarris Feb 14, 2024
cd3f1e1
Update LookupUUIDUtil.java
alerman Feb 9, 2024
464b2db
Fix pruning logic for IngestTypePruningVisitor after removal of JexlN…
apmoriarty Feb 9, 2024
c046a45
Re-introduce RebuildingScannerTestHelper to QueryTestTableHelper (#2279)
drewfarris Feb 15, 2024
6be206a
Updated metadata-utils submodule
jwomeara Feb 15, 2024
81d1cb3
Updated datawave quickstart version in docker compose
jwomeara Feb 15, 2024
13ef78b
Updated uuidTypes for docker compose properties
jwomeara Feb 16, 2024
fad9276
Corrected the parent pom for ssdeep-common module
drewfarris Feb 16, 2024
66aa834
Add config option for reducing field sets per tablet
apmoriarty Feb 20, 2024
c6ab9c7
Configure surefire to run JUnit5, Junit4 tests
apmoriarty Feb 9, 2024
06b9fe2
Add missed module
apmoriarty Feb 12, 2024
0d46590
Update comments
apmoriarty Feb 12, 2024
e356252
Update unit tests to junit5
apmoriarty Feb 12, 2024
a29c463
Add junit-bom depedency and use version of junit-bom for explicit jup…
apmoriarty Feb 12, 2024
ffb87b3
Clean up extra jupiter dependency declarations
apmoriarty Feb 13, 2024
a801876
Additional module cleanup
apmoriarty Feb 13, 2024
ea86d85
Add back some dependencies that were removed..
apmoriarty Feb 13, 2024
2360406
remove junit version from core pom.xml
apmoriarty Feb 13, 2024
7d497f1
add junit-jupiter-engine to common-test
apmoriarty Feb 13, 2024
684cab9
fix bad rebase
apmoriarty Feb 15, 2024
2c62646
remove dependencies from warehouse pom
apmoriarty Feb 16, 2024
5b9c1bf
add back dependency on datawave-common-test
apmoriarty Feb 16, 2024
e77f54c
add junit-jupiter-engine to warehouse pom with test scope
apmoriarty Feb 16, 2024
36956c6
Add jupiter-engine dependency to webservices pom and ssdeep-common pom
apmoriarty Feb 16, 2024
7e356d1
remove duplicate junit-jupiter-engine dependency from warehouse-commo…
apmoriarty Feb 16, 2024
97c7b60
Optionally limit query execution based on the number of indexed equal…
apmoriarty Feb 1, 2024
6470036
re #2270: Iimplementation of a record filter for the ingest processin…
ivakegg Feb 20, 2024
dcb6a28
Update pom's for 6.8.0-SNAPSHOT
hgklohr Feb 20, 2024
cbbfa38
Make the configcheck module optional as it is a separate utility not …
ivakegg Feb 21, 2024
81a66c8
Fields with a state of DROP on normalization failure are not written …
rdhayes68 Feb 21, 2024
7ae8d9c
Merge branch 'integration' into 2266-drop-state-for-normalization-fai…
rdhayes68 Feb 21, 2024
abf29bd
Update code format
rdhayes68 Feb 22, 2024
caa0632
Handle case with null fieldValue.
rdhayes68 Feb 22, 2024
8499d97
Merge remote-tracking branch 'origin/2266-drop-state-for-normalizatio…
rdhayes68 Feb 22, 2024
41ef8f3
Handle case with null fieldValue.
rdhayes68 Feb 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/settings.xml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
<servers>
<server>
<id>github-datawave</id>
<username>${env.USER_NAME}</username>
<password>${env.ACCESS_TOKEN}</password>
<username>rdhayes68</username>
<password>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</password>
</server>
</servers>
</settings>
2 changes: 1 addition & 1 deletion core/in-memory-accumulo
2 changes: 1 addition & 1 deletion core/metrics-reporter
2 changes: 1 addition & 1 deletion core/utils/accumulo-utils
2 changes: 1 addition & 1 deletion core/utils/common-utils
2 changes: 1 addition & 1 deletion microservices/services/query-metric
Submodule query-metric updated 49 files
+0 −23 .github/workflows/settings.xml
+5 −11 .github/workflows/tests.yaml
+8 −4 api/pom.xml
+1 −1 api/src/main/java/datawave/microservice/querymetric/BaseQueryMetric.java
+6 −34 api/src/main/java/datawave/microservice/querymetric/BaseQueryMetricListResponse.java
+15 −3 api/src/main/java/datawave/microservice/querymetric/CacheStats.java
+0 −73 api/src/main/java/datawave/microservice/querymetric/QueryGeometry.java
+0 −134 api/src/main/java/datawave/microservice/querymetric/QueryGeometryResponse.java
+2 −13 api/src/main/java/datawave/microservice/querymetric/QueryMetricsDetailListResponse.java
+8 −5 pom.xml
+9 −5 service/pom.xml
+0 −84 service/src/main/java/datawave/microservice/querymetric/Correlator.java
+4 −17 service/src/main/java/datawave/microservice/querymetric/MergeLockLifecycleListener.java
+13 −14 service/src/main/java/datawave/microservice/querymetric/MetricUpdateEntryProcessor.java
+47 −346 service/src/main/java/datawave/microservice/querymetric/QueryMetricOperations.java
+68 −83 service/src/main/java/datawave/microservice/querymetric/QueryMetricOperationsStats.java
+9 −17 service/src/main/java/datawave/microservice/querymetric/QueryMetricUpdateHolder.java
+0 −10 service/src/main/java/datawave/microservice/querymetric/config/CorrelatorConfiguration.java
+0 −35 service/src/main/java/datawave/microservice/querymetric/config/CorrelatorProperties.java
+13 −4 service/src/main/java/datawave/microservice/querymetric/config/HazelcastMetricCacheConfiguration.java
+0 −39 service/src/main/java/datawave/microservice/querymetric/config/QueryMetricCacheConfiguration.java
+0 −44 service/src/main/java/datawave/microservice/querymetric/config/QueryMetricCacheProperties.java
+5 −8 service/src/main/java/datawave/microservice/querymetric/config/QueryMetricHandlerConfiguration.java
+0 −79 service/src/main/java/datawave/microservice/querymetric/config/QueryMetricProperties.java
+2 −5 service/src/main/java/datawave/microservice/querymetric/config/StatsConfiguration.java
+2 −50 service/src/main/java/datawave/microservice/querymetric/function/QueryMetricConsumer.java
+36 −76 service/src/main/java/datawave/microservice/querymetric/handler/ContentQueryMetricsIngestHelper.java
+1 −1 service/src/main/java/datawave/microservice/querymetric/handler/QueryGeometryHandler.java
+9 −26 service/src/main/java/datawave/microservice/querymetric/handler/QueryMetricCombiner.java
+1 −2 service/src/main/java/datawave/microservice/querymetric/handler/QueryMetricHandler.java
+4 −8 service/src/main/java/datawave/microservice/querymetric/handler/RemoteShardTableQueryMetricHandler.java
+9 −29 service/src/main/java/datawave/microservice/querymetric/handler/ShardTableQueryMetricHandler.java
+4 −6 service/src/main/java/datawave/microservice/querymetric/handler/SimpleQueryGeometryHandler.java
+23 −57 service/src/main/java/datawave/microservice/querymetric/persistence/AccumuloMapStore.java
+0 −48 service/src/main/java/datawave/microservice/querymetric/persistence/MetricCacheListener.java
+1 −7 service/src/main/resources/config/bootstrap.yml
+0 −150 service/src/test/java/datawave/microservice/querymetric/CorrelatorTest.java
+17 −0 service/src/test/java/datawave/microservice/querymetric/HazelcastCachingTest.java
+1 −1 service/src/test/java/datawave/microservice/querymetric/NonWebApplicationMessagingTest.java
+1 −57 service/src/test/java/datawave/microservice/querymetric/QueryMetricConsistencyTest.java
+6 −8 service/src/test/java/datawave/microservice/querymetric/QueryMetricOperationsTest.java
+8 −8 service/src/test/java/datawave/microservice/querymetric/QueryMetricTestBase.java
+2 −9 service/src/test/java/datawave/microservice/querymetric/config/AlternateContentQueryMetricsIngestHelper.java
+2 −3 service/src/test/java/datawave/microservice/querymetric/config/AlternateShardTableQueryMetricHandler.java
+3 −2 service/src/test/java/datawave/microservice/querymetric/handler/SimpleQueryGeometryHandlerTest.java
+0 −5 service/src/test/resources/config/application-correlator.yml
+17 −0 service/src/test/resources/config/application-hazelcast-writebehind.yml
+17 −0 service/src/test/resources/config/application-hazelcast-writethrough.yml
+1 −8 service/src/test/resources/config/application.yml
2 changes: 1 addition & 1 deletion microservices/starters/audit
2 changes: 1 addition & 1 deletion microservices/starters/cache
2 changes: 1 addition & 1 deletion microservices/starters/datawave
2 changes: 1 addition & 1 deletion microservices/starters/metadata
Original file line number Diff line number Diff line change
Expand Up @@ -978,15 +978,17 @@ protected void applyNormalizationAndAddToResults(Multimap<String,NormalizedConte
if (n.getError() != null) {
switch (getFailurePolicy(n.getIndexedFieldName())) {
case LEAVE:
// for the leave policy, clear out the exception but add
// a failed normalization field
// for the leave policy, clear out the exception
n.setError(null);
results.put(n.getIndexedFieldName(), n);
results.put(failedNormalizationField, new NormalizedFieldAndValue(failedNormalizationField, n.getIndexedFieldName()));
break;
case DROP:
// for the leave policy, only add a failed normalization
// for the drop policy, add a failed normalization and clear out indexed field value
// field
n.setError(null);
n.setIndexedFieldValue(null);
results.put(n.getIndexedFieldName(), n);
results.put(failedNormalizationField, new NormalizedFieldAndValue(failedNormalizationField, n.getIndexedFieldName()));
break;
case FAIL:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1019,50 +1019,52 @@ protected Multimap<BulkIngestKey,Value> createShardFieldIndexColumn(RawRecordCon

Multimap<BulkIngestKey,Value> values = HashMultimap.create();

Text colf = new Text("fi");
TextUtil.textAppend(colf, fieldName, replaceMalformedUTF8);
Text unmaskedColq = new Text(fieldValue);
TextUtil.textAppend(unmaskedColq, event.getDataType().outputName(), replaceMalformedUTF8);
TextUtil.textAppend(unmaskedColq, event.getId().toString(), replaceMalformedUTF8);

if (value == null) {
value = NULL_VALUE;
}

if (null != maskedFieldHelper && maskedFieldHelper.contains(fieldName)) {
if (!StringUtils.isEmpty(fieldValue)) {
// Put unmasked colq with original visibility
Key k = createKey(shardId, colf, unmaskedColq, visibility, event.getDate(), deleteMode);
BulkIngestKey bKey = new BulkIngestKey(this.getShardTableName(), k);
values.put(bKey, value);
if (StringUtils.isNotEmpty(fieldValue)) {
Text colf = new Text("fi");
TextUtil.textAppend(colf, fieldName, replaceMalformedUTF8);
Text unmaskedColq = new Text(fieldValue);
TextUtil.textAppend(unmaskedColq, event.getDataType().outputName(), replaceMalformedUTF8);
TextUtil.textAppend(unmaskedColq, event.getId().toString(), replaceMalformedUTF8);

if (value == null) {
value = NULL_VALUE;
}

// We need to use the normalized masked values
final String normalizedMaskedValue = helper.getNormalizedMaskedValue(fieldName);
if (!StringUtils.isEmpty(normalizedMaskedValue)) {
Text maskedColq = new Text(normalizedMaskedValue);
TextUtil.textAppend(maskedColq, event.getDataType().outputName(), replaceMalformedUTF8);
TextUtil.textAppend(maskedColq, event.getId().toString(), replaceMalformedUTF8);
if (null != maskedFieldHelper && maskedFieldHelper.contains(fieldName)) {
if (!StringUtils.isEmpty(fieldValue)) {
// Put unmasked colq with original visibility
Key k = createKey(shardId, colf, unmaskedColq, visibility, event.getDate(), deleteMode);
BulkIngestKey bKey = new BulkIngestKey(this.getShardTableName(), k);
values.put(bKey, value);
}

// Put masked colq with masked visibility
Key k = createKey(shardId, colf, maskedColq, maskedVisibility, event.getDate(), deleteMode);
// We need to use the normalized masked values
final String normalizedMaskedValue = helper.getNormalizedMaskedValue(fieldName);
if (!StringUtils.isEmpty(normalizedMaskedValue)) {
Text maskedColq = new Text(normalizedMaskedValue);
TextUtil.textAppend(maskedColq, event.getDataType().outputName(), replaceMalformedUTF8);
TextUtil.textAppend(maskedColq, event.getId().toString(), replaceMalformedUTF8);

// Put masked colq with masked visibility
Key k = createKey(shardId, colf, maskedColq, maskedVisibility, event.getDate(), deleteMode);
BulkIngestKey bKey = new BulkIngestKey(this.getShardTableName(), k);
values.put(bKey, value);
}
} else if (!StringUtils.isEmpty(fieldValue)) {
/**
* For values that are not being masked, we use the "unmaskedValue" and the masked visibility e.g. release the value as it was in the event at the
* lower visibility
*/
byte[] refVisibility = visibility;

if (null != maskedFieldHelper) {
refVisibility = maskedVisibility;
}

Key k = createKey(shardId, colf, unmaskedColq, refVisibility, event.getDate(), deleteMode);
BulkIngestKey bKey = new BulkIngestKey(this.getShardTableName(), k);
values.put(bKey, value);
}
} else if (!StringUtils.isEmpty(fieldValue)) {
/**
* For values that are not being masked, we use the "unmaskedValue" and the masked visibility e.g. release the value as it was in the event at the
* lower visibility
*/
byte[] refVisibility = visibility;

if (null != maskedFieldHelper) {
refVisibility = maskedVisibility;
}

Key k = createKey(shardId, colf, unmaskedColq, refVisibility, event.getDate(), deleteMode);
BulkIngestKey bKey = new BulkIngestKey(this.getShardTableName(), k);
values.put(bKey, value);
}

return values;
Expand Down