Sanitize log names

[chunyong-lin]

to: @ryandeivert @Ryxias @blakemotl
cc: @airbnb/streamalert-maintainers
related to:
resolves: #1186

Background

See the issue #1186 for more details.

Changes

• Replace all . (dot) in the log names in conf/schemas/*.json or conf/logs.json with _ (underscore). The change is made to Config class where all the conf files are loaded.
• Also update rule test method to replace . with _ when inspect log source in the testing events. Eventually there are two carbonblack testing events with log set to carbonblack:ingress.event.procstart and carbonblack:ingress_event_procstart.

Testing

• Add unit tests to reproduce the issue and test the code change.
• python manage.py athena create-table --bucket nobody2020030420-streamalert-data --table-name carbonblack_alert_status_updated works in staging environment with the fix.

[coveralls]

Coverage increased (+0.008%) to 95.424% when pulling a3ea5c5 on sanitize_log_names into 1b704c6 on release-3-1-0.

[ryandeivert]

eeek conditional blocks like this make me cringe so hard. readability is so bad. can be break this up into multiple statements?

like:

infra_config = config.get('global', {}).get('infrastructure')
if not infra_config:
    return config

if 'firehose' not in infra_config:
    return config

# ... and so on ...

[Ryxias]

Alternative way is to lump it into a function>

if self._requires_sanitized_log_names(etc):
  return config['global']....

return config

[chunyong-lin]

Hi @ryandeivert , about your concern

does theis _sanitize_logs_name function actually replace the value within a person's config to be written back out to disk??

if so, I don't think we should do this... this could mess up rules, etc that use specific logs. I think and "transformation" for log name --> table name + firehose name should be done by us under the hood

I have checked the code and tested the new changes in the staging environment with a weird log name cloudwatch:events.dots-test, the person's config on the disk is not changed, classifier, rules engine, alert processor and athena partition lambda functions are working as expected. Can you PTAL my latest two commits which address the comments. Thank you thank you.