[TI_Anomali] Fix transform sort order field #7000

kcreddy · 2023-07-18T16:50:37Z

What does this PR do?

Existing sort order for latest transform is event.ingested. Since this is not precise to the millisecond level, whenever events comes with both deleted and added for same indicator (within a second), the last activity is not preserved and the indicator is simply deleted.
- This PR changes the sort order and the sync fields to @timestamp which has millisecond resolution, and thus preserves the correct order of events to store in the destination indices.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Relates [TI_Anomali] Updated indicators are deleted in destination indices #7012

elasticmachine · 2023-07-18T17:17:33Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-19T17:11:35.331+0000
Duration: 21 min 46 sec

Test stats 🧪

Test	Results
Failed	1
Passed	10
Skipped	0
Total	11

Test errors

Expand to view the tests failures

`pipeline test: test-anomali-threatstream.json – ti_anomali.threatstream`

Expand to view the error details

 null

Expand to view the stacktrace

 test case failed: Expected results are different from actual ones: --- want
+++ got
@@ -1,6 +1,7 @@
 {
     "expected": [
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -65,6 +66,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -127,6 +129,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -191,6 +194,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -245,6 +249,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -309,6 +314,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -373,6 +379,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -436,6 +443,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -488,6 +496,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -550,6 +559,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -614,6 +624,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -669,6 +680,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -724,6 +736,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -787,6 +800,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -842,6 +856,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -898,6 +913,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -953,6 +969,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1009,6 +1026,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1065,6 +1083,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1121,6 +1140,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1177,6 +1197,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1232,6 +1253,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1291,6 +1313,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1345,6 +1368,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1407,6 +1431,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1463,6 +1488,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1519,6 +1545,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1576,6 +1603,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1629,6 +1657,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1690,6 +1719,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1752,6 +1782,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1808,6 +1839,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1865,6 +1897,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1921,6 +1954,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -1977,6 +2011,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2031,6 +2066,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2087,6 +2123,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2142,6 +2179,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2197,6 +2235,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2259,6 +2298,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2313,6 +2353,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2367,6 +2408,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2423,6 +2465,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2486,6 +2529,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2543,6 +2587,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2605,6 +2650,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2666,6 +2712,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2722,6 +2769,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2776,6 +2824,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2832,6 +2881,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2887,6 +2937,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -2950,6 +3001,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3004,6 +3056,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3058,6 +3111,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3112,6 +3166,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3173,6 +3228,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3227,6 +3283,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3283,6 +3340,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3338,6 +3396,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3393,6 +3452,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3448,6 +3508,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3511,6 +3572,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3566,6 +3628,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3622,6 +3685,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3677,6 +3741,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3732,6 +3797,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3788,6 +3854,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3845,6 +3912,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3898,6 +3966,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -3960,6 +4029,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4015,6 +4085,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4069,6 +4140,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4123,6 +4195,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4185,6 +4258,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4239,6 +4313,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4294,6 +4369,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4348,6 +4424,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4411,6 +4488,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4466,6 +4544,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4522,6 +4601,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4589,6 +4669,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4652,6 +4733,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4706,6 +4788,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4764,6 +4847,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4826,6 +4910,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4888,6 +4973,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -4956,6 +5042,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5019,6 +5106,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5078,6 +5166,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5138,6 +5227,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5199,6 +5289,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5261,6 +5352,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5322,6 +5414,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5380,6 +5473,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5444,6 +5538,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5492,6 +5587,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5542,6 +5638,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "added_at": "2020-10-08T12:22:11.000Z",
@@ -5590,6 +5687,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "classification": "public",
@@ -5640,6 +5738,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "classification": "private",
@@ -5689,6 +5788,7 @@
             }
         },
         {
+            "@timestamp": "2020-10-08T12:22:11.000Z",
             "anomali": {
                 "threatstream": {
                     "classification": "private",

Steps errors

Expand to view the steps failures

`Test integration: ti_anomali`

Took 3 min 37 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Google Storage Download`

Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-18T17:17:42Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚 10.526
Classes	100.0% (`0/0`)	💚 10.526
Methods	100.0% (`4/4`)	💚 15.236
Lines	100.0% (`0/0`)	💚 14.331
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-07-19T10:30:02Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2023-07-19T11:29:33Z

Where does the @timestamp value originate in Anomali? Is from Anamoli or is it added by Filebeat? Filebeat might create timestamps that are equal across two events (given ms precision).

kcreddy · 2023-07-19T11:34:15Z

Hey @andrewkroh,
Its not set inside the pipeline, hence filebeat adds them. But the SDK itself parses the events one after the other, meaning first deleted event are parsed, then sent to agent. Then added event is parsed and sent.
Do you think it would even cause the issue? If so, I will try to modify the ingest pipeline to use the timestamps being generated in the SDK.

Update:
I see even SDK only generates only 1 timestamp for both deleted and added messages. I can update the SDK code quick if you think timestamps should come from there.

kcreddy · 2023-07-19T17:16:49Z

@andrewkroh, I made the timestamp changes based on your suggestion. I am now assigning the timestamp created from within the Anomali SDK to @timestamp field.
Here's the updated SDK PR: https://github.com/elastic/filebeat-anomali-integrator-sdk/pull/17/files where I am using different timestamp for deleted event and added event i.e., added event is 1s after the deleted event.

I am using that calculated timestamp to populate @timestamp field inside the ingest pipeline, so that added events are always after the deleted events. I also tested the functioning by adding, deleting, and updating indicators. These scenarios are working fine.

elasticmachine · 2023-07-31T07:55:05Z

Package ti_anomali - 1.14.1 containing this change is available at https://epr.elastic.co/search?package=ti_anomali

* Change sort order field * for test * update pr num * remove default * change fleet version to upgrade the transform * add timestamp field * update pipeline tests

kcreddy added 2 commits July 18, 2023 21:20

Change sort order field

f9bfa7c

for test

016aeb9

kcreddy added the bug Something isn't working label Jul 18, 2023

kcreddy self-assigned this Jul 18, 2023

kcreddy added the Integration:Anomali label Jul 18, 2023

kcreddy added 2 commits July 19, 2023 12:28

update pr num

5c8a7be

remove default

fac6bf4

kcreddy marked this pull request as ready for review July 19, 2023 10:26

kcreddy requested a review from a team as a code owner July 19, 2023 10:26

kcreddy added the Team:Security-External Integrations label Jul 19, 2023

change fleet version to upgrade the transform

94ad25d

kcreddy added 2 commits July 19, 2023 20:53

add timestamp field

928c413

update pipeline tests

34e761e

kcreddy requested a review from andrewkroh July 20, 2023 09:45

andrewkroh approved these changes Jul 26, 2023

View reviewed changes

kcreddy merged commit 3ce2c0b into elastic:main Jul 31, 2023
4 checks passed

kcreddy deleted the anomali_sort_order branch July 31, 2023 07:51

gizas pushed a commit that referenced this pull request Sep 5, 2023

[TI_Anomali] Fix transform sort order field (#7000)

0944c68

* Change sort order field * for test * update pr num * remove default * change fleet version to upgrade the transform * add timestamp field * update pipeline tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TI_Anomali] Fix transform sort order field #7000

[TI_Anomali] Fix transform sort order field #7000

kcreddy commented Jul 18, 2023 •

edited

elasticmachine commented Jul 18, 2023 •

edited

Build stats

Test stats 🧪

`pipeline test: test-anomali-threatstream.json – ti_anomali.threatstream`

`Test integration: ti_anomali`

`Google Storage Download`

elasticmachine commented Jul 18, 2023 •

edited

elasticmachine commented Jul 19, 2023

andrewkroh commented Jul 19, 2023

kcreddy commented Jul 19, 2023 •

edited

kcreddy commented Jul 19, 2023 •

edited

elasticmachine commented Jul 31, 2023

[TI_Anomali] Fix transform sort order field #7000

[TI_Anomali] Fix transform sort order field #7000

Conversation

kcreddy commented Jul 18, 2023 • edited

What does this PR do?

Checklist

Related issues

elasticmachine commented Jul 18, 2023 • edited

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

pipeline test: test-anomali-threatstream.json – ti_anomali.threatstream

Steps errors

Test integration: ti_anomali

Google Storage Download

🤖 GitHub comments

elasticmachine commented Jul 18, 2023 • edited

🌐 Coverage report

elasticmachine commented Jul 19, 2023

andrewkroh commented Jul 19, 2023

kcreddy commented Jul 19, 2023 • edited

kcreddy commented Jul 19, 2023 • edited

elasticmachine commented Jul 31, 2023

kcreddy commented Jul 18, 2023 •

edited

elasticmachine commented Jul 18, 2023 •

edited

`pipeline test: test-anomali-threatstream.json – ti_anomali.threatstream`

`Test integration: ti_anomali`

`Google Storage Download`

elasticmachine commented Jul 18, 2023 •

edited

kcreddy commented Jul 19, 2023 •

edited

kcreddy commented Jul 19, 2023 •

edited