[TI_Anomali] Fix destination fields mapping conflicts

[kcreddy]

What does this PR do?

Field mapping conflicts exists between source and destination indices (created by transform) of Anomali. The issue happened because not all the files in the elasticsearch/transform/latest_ioc/fields folder is considered as destination fields. Only the fields inside the fields.yml are taken into the mapping. Hence dynamic mapping is being applied for rest of the fields.

• This PR removes the additional files from elasticsearch/transform/latest_ioc/fields folder and and copies corresponding fields into elasticsearch/transform/latest_ioc/fields/fields.yml

NOTE: Due to issue in elastic-package not expanding the ECS fields under elasticsearch/transform/latest_ioc/fields/fields.yml mapped with - external: ecs definition, the ECS fields must be manually copied from build/packages/ti_anomali/<VERSION>/data_stream/threatstream/fields/ecs.yml
Reference Issue - elastic/elastic-package#1369

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-08-01T05:29:45.253+0000

• Duration: 18 min 36 sec

Test stats 🧪

Test	Results
Failed	0
Passed	11
Skipped	0
Total	11

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (1/1)	💚
Files	100.0% (1/1)	💚 2.828
Classes	100.0% (1/1)	💚 2.828
Methods	100.0% (13/13)	💚 7.707
Lines	91.071% (357/392)	👍 0.091
Conditionals	100.0% (0/0)	💚

[andrewkroh]

Due to issue in elastic-package not expanding the ECS fields

Can you please link to the associated elastic-package bug issue in the PR description.

[kcreddy]

Added issue to the description

[andrewkroh]

Does an issue already exist for elastic/kibana about Fleet only reading the contents of elasticsearch/transform/{name}/fields/fields.yml instead of elasticsearch/transform/{name}/fields/*? If not, can you please create a bug issue for that too. IMO the package-spec is clear that it should honor any fields in the fields/ directory not just fields.yml.

https://github.com/elastic/package-spec/blob/991ae115d7ac50175902072d2e3a4bb685f36c02/spec/integration/elasticsearch/transform/spec.yml#L11-L12

https://github.com/elastic/package-spec/blob/991ae115d7ac50175902072d2e3a4bb685f36c02/spec/integration/elasticsearch/transform/spec.yml#L51-L55

[kcreddy]

IMO the package-spec is clear that it should honor any fields in the fields/ directory not just fields.yml

Yeah, I want to test it again to see if the problem is only the defined external fields are not getting expanded, or if multiple files not being honoured. I will update here.

[andrewkroh]

Somewhere (docs or changelog) I think we should mention that users should delete the logs-ti_anomali_latest.threatstream-1 after upgrading. (That is until we can automatically cleanup the old index).

[kcreddy]

Makes sense.

[kcreddy]

Updated the changelog to redirect to the docs. Added information about deleting older index and why we are doing it. Wording is generic enough to make users delete any future older versions. We could remove this doc section once the auto-cleanup is available.

[kcreddy]

Does an issue already exist for elastic/kibana about Fleet only reading the contents of elasticsearch/transform/{name}/fields/fields.yml instead of elasticsearch/transform/{name}/fields/*?

I confirmed that multiple fields definition files inside transform folder is not working. I created the issue inside elastic-package: elastic/kibana#162808

[elasticmachine]

Package ti_anomali - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=ti_anomali