METRON-1608: Add configuration for threat.triage.field name

[merrimanr]

Contributor Comments

This PR adds a configuration to the global config for the threat.triage.score field name, similar to what was done in #1010 for source.type, minus the UI changes.

I also opportunistically fixed a bug where the source.type field name wasn't being read from the global config. Normally this would be a separate PR but the fix overlaps with what was done here so I included it. I also added a constant for the source.type.field property.

Changes

• Added constants for source.type.field and threat.triage.field (would threat.triage.score.field be better?)
• Added getter/setter for AccessConfig in the ElasticsearchDao (necessary for testing)
• Refactored the default threat triage field name in ElasticsearchMetaAlertDao to match source type field name pattern
• Added a getField method that gets the field name from the global config or returns the default field name if not found
• Added unit and integration tests for both source.type.field and threat.triage.field

Testing

Testing this in full dev requires changing our indexing topology to use fields with '.'s rather than ':'s. To do that:

1. Stop the snort sensor with service sensor-stubs stop snort
2. Delete the snort ES index with curl -XDELETE http://node1:9200/snort_index*
3. Edit the template at /var/lib/ambari-agent/cache/common-services/METRON/0.5.0/package/files/snort_index.template by changing source:type to source.type and threat:triage:*score to threat.triage.*score
4. Reinstall the template using the "Elasticsearch Template Install" in Ambari
5. Update the global config with the new field names:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "es.clustername": "metron",
  "es.ip": "node1:9300",
  "es.date.format": "yyyy.MM.dd.HH",
  "parser.error.topic": "indexing",
  "update.hbase.table": "metron_update",
  "update.hbase.cf": "t",
  "es.client.settings": {
    "client.transport.ping_timeout": "500s"
  },
  "profiler.client.period.duration": "15",
  "profiler.client.period.duration.units": "MINUTES",
  "user.settings.hbase.table": "user_settings",
  "user.settings.hbase.cf": "cf",
  "bootstrap.servers": "node1:6667",
  "geo.hdfs.file": "/apps/metron/geo/default/GeoLite2-City.mmdb.gz",
"source.type.field":"source.type",
"threat.triage.score.field":"threat.triage.score"
}' 'http://node1:8082/api/v1/global/config'

6. Update the snort indexing config to not DEDOT the field names:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "hdfs": {
    "index": "snort",
    "batchSize": 1,
    "enabled": true
  },
  "elasticsearch": {
    "index": "snort",
    "batchSize": 1,
    "enabled": true,
    "fieldNameConverter": "NOOP"
  },
  "solr": {
    "index": "snort",
    "batchSize": 1,
    "enabled": true
  }
}' 'http://node1:8082/api/v1/sensor/indexing/config/snort'

7. Start the snort sensor again with service sensor-stubs start snort
8. Wait for data to appear in the snort ES index and then perform a search:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"facetFields":[],
  "from": 0,
  "indices": [
    "snort"
  ],
  "query": "*",
  "size": 5
}' 'http://node1:8082/api/v1/search/search'

9. Grab a guid from one of the snort search results and create a metaalert:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "alerts": [
    {
      "guid": "1c082f1a-ad3a-4e46-ae1e-828b73c9a016",
      "sensorType": "snort"
    }
  ],
  "groups": [
    "string"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'

10. Now find the metaalert from the guid returned in the previous step:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "guid": "1ad8be85-3164-4be9-a773-379abb044d0a",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'

Before this PR there were a couple problems. First the metaalert had the source:type field even though we've switched to "source.type" in the global config. Second, the threat.triage.score field in the metaalert would be 0 because the wrong threat triage field name is used to get scores from alerts. With this PR the metalaert should correctly have the source.type field and the threat.triage.score field should be > 0 (assuming the snort alert has a threat triage score):

{
  "average": 10,
  "max": 10,
  "threat.triage.score": 10,
  "count": 1,
  "groups": [
    "string"
  ],
  "sum": 10,
  "source.type": "metaalert",
  "min": 10,
  "median": 10,
  "alert": [
    ...
  ]
}

Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

For all changes:

• Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
• Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
• Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

• Have you included steps to reproduce the behavior or problem that is being changed or addressed?

• Have you included steps or a guide to how the change may be verified and tested manually?

• Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:

  mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh 
  

• Have you written or updated unit tests and or integration tests to verify your changes?

• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

• Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:

  cd site-book
  mvn site
  

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

[nickwallen]

It seems like we have duplicated this logic here and in SearchServiceImpl. Would it make sense to extract this out?

[merrimanr]

The latest commit extracts this to ConfigurationsUtils class in metron-common. This class is already really big so I was hesitant to add it there but I'm not sure where else it belongs.

[nickwallen]

Just another point of duplication. We define what the default value is here and in ElasticsearchMetaAlertDao. Could be extracted out along with the other bits.

[merrimanr]

This one is tricky. We can't just reference ElasticsearchMetaAlertDao.SOURCE_TYPE because metron-elasticsearch is a runtime dependency in metron-rest. The only other solution I can think of is to define a second constant in Constants but I think that is confusing since it's specific to ES. I remember @justinleet working through this in another, maybe he can chime in.

[nickwallen]

FYI - There is also another source.type.field reference that would be an easy update here

[merrimanr]

Good feedback @nickwallen. The latest commit should satisfy your comments except the duplicated SOURCE_TYPE issue. Not sure what the best approach is so I'm inclined to leave it alone.

[nickwallen]

+1 looks good. Thanks

I wouldn't worry about that constant. Not worth it.