Fix get mappings view API incorrectly returning ecs path

[jowg-amazon]

Description

Currently the get mappings view api incorrectly returns the ecs path for logs with ocsf or raw field types if a custom rule is created using a raw field. This is because whenever a new rule is created, the rule field(s) gets indexed into .opensearch-sap-log-types-config index along with the log_type (ex. below). However, because there are already preset mappings defined for certain log types like cloudtrail, the raw_field, ecs, and ocsf mappings are already defined in this index. Then when the get Mappings view API is called, the new document from the custom rule overwrites the existing predefined mapping which causes the ecs format to be incorrectly returned.

{
        "_index": ".opensearch-sap-log-types-config",
        "_id": "aws.cloudtrail.event_name|",
        "_score": 1,
        "_source": {
          "raw_field": "aws.cloudtrail.event_name",
          "log_types": [
            "cloudtrail"
          ]
        }
}

This PR performs additional logic to check whether or not the raw field from the custom rule already exists in the existingFieldMappings (i.e. if this is a known mapping that is predefined in the index). If it is defined, then it doesn't add a the new mapping into the .opensearch-sap-log-types-config index.

The integ test ensures that the mappings view API returns the same output before and after a custom rule with a raw field is created to verify that the correct path is returned.

Issues Resolved

#866

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[engechas]

Wasn't added by you so optional - Maybe existingFieldMapping instead of e. e reads as Exception to me

[engechas]

Is there a possibility of false positives here? Say I have a field with name field2 and then another field with name field. If field2 is evaluated first, the contains call would match for field even though it's an entry for field2

[jowg-amazon]

Yeah I agree e is a bit confusing. I changed it to existingFieldMapping. Also yes that false positive is a possibility. I changed the logic so that we grab the right side of the ID of the doc which corresponds to the ecs field name when we load in our predefined mappings. That way we can do an equals check instead to ensure that there are no duplicates.

ex. of document from pre-defined mapping

{
        "_index": ".opensearch-sap-log-types-config",
        "_id": "eventName|aws.cloudtrail.event_name",
        "_score": 1,
        "_source": {
          "raw_field": "eventName",
          "ecs": "aws.cloudtrail.event_name",
          "ocsf": "api.operation",
          "log_types": [
            "s3",
            "cloudtrail"
          ]
        }
      }

[codecov]

Codecov Report

Attention: Patch coverage is 58.33333% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 24.79%. Comparing base (75c4429) to head (09cf6ab).

❗ Current head 09cf6ab differs from pull request most recent head ff4f63b. Consider uploading reports for the commit ff4f63b to get more accurate results

Files	Patch %	Lines
...arch/securityanalytics/logtype/LogTypeService.java	58.33%	1 Missing and 4 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #867      +/-   ##
============================================
- Coverage     24.97%   24.79%   -0.19%     
+ Complexity     1043     1026      -17     
============================================
  Files           277      277              
  Lines         12771    12723      -48     
  Branches       1391     1403      +12     
============================================
- Hits           3190     3155      -35     
+ Misses         9307     9300       -7     
+ Partials        274      268       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[engechas]

Is this backwards compatible? Wondering if we will have duplicate mappings for a mapping that was generated before the pipe system was introduced (say an upgrade from an older to the current version).

I am also not very familiar with this part of the code though, so if that is not a concern, then I am good to approve.

[jowg-amazon]

Backwards compatibility should be fine because the mapping should not have been inserted in the first place but now we're ensuring that it doesn't get generated. If they encounter this bug though I think they may need to delete the log-type-config index so that we can automatically regenerate it with the correct mappings.

[eirsep]

@sbcd90 can we change logic to remove the code for adding any new field into this index at all during custom rule creation?