Domains ending in '.loc' on blacklists break OSXCollector's Analyze Filter

[bfrizb]

Domains ending in '.loc' on blacklists break OSXCollector's Analyze Filter

For example, if the whitelist has "yelp.loc" on it and you run the analyze filter:
$ cat MALWARE-TEST-/MALWARE-TEST- | python -m osxcollector.output_filters.analyze

you'll get a traceback list this:

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/analyst/Desktop/osxcollector-master/osxcollector/output_filters/analyze.py", line 453, in <module>
    main()
  File "/Users/analyst/Desktop/osxcollector-master/osxcollector/output_filters/analyze.py", line 449, in main
    run_filter_main(AnalyzeFilter)
  File "osxcollector/output_filters/base_filters/output_filter.py", line 128, in run_filter_main
    filter_arguments = output_filter_cls().get_argument_parser()
  File "/Users/analyst/Desktop/osxcollector-master/osxcollector/output_filters/analyze.py", line 81, in __init__
    filter_chain.append(OpenDnsRelatedDomainsFilter(related_when=AnalyzeFilter.find_related_when, **kwargs))
  File "osxcollector/output_filters/opendns/related_domains.py", line 64, in __init__
    self._whitelist = create_blacklist(config_get_deep('domain_whitelist'))
  File "osxcollector/output_filters/util/blacklist.py", line 37, in create_blacklist
    return Blacklist(blacklist_name, blacklist_keys, blacklist_file_path, blacklist_is_regex, blacklist_is_domains)
  File "osxcollector/output_filters/util/blacklist.py", line 66, in __init__
    self._blacklisted_values = [self._convert_to_matching_term(val) for val in self._blacklisted_values]
  File "osxcollector/output_filters/util/blacklist.py", line 87, in _convert_to_matching_term
    domain = clean_domain(blacklisted_value)
  File "osxcollector/output_filters/util/domains.py", line 54, in clean_domain
    raise BadDomainError(u'Can not clean {0} {1}'.format(unclean_domain, repr(extracted)))
osxcollector.output_filters.exceptions.BadDomainError: Can not clean yelp.loc ExtractResult(subdomain=u'yelp', domain=u'loc', suffix='')

[ivanlei]

@bfrizb should we really fix this? The domain can't be cleaned so it's not compatible with the blacklist filter. Seems like it's better to warn the user and fail then to succeed and silently drop something.

[bfrizb]

It's a minor point, but maybe the error message could be a little clearer? When I encountered it, I had to dig around in the code to figure out that ".loc" was the problem part of the domain.

[ivanlei]

I think an error that shows:

raise BadDomainError(u'Can not clean {0} {1}'.format(unclean_domain, repr(extracted)))
osxcollector.output_filters.exceptions.BadDomainError: Can not clean yelp.loc ExtractResult(subdomain=u'yelp', domain=u'loc', suffix='')

is pretty sufficient in explaining things.