Add ScanMalware expansion module - #803
Conversation
Queries the public ScanMalware API for a domain, hostname, URL or IP attribute. No API key is required: the API is anonymous, and a key only raises the rate limit. For a domain or hostname it returns two different populations, subdomains seen in Certificate Transparency DNS records and hosts a browser actually resolved and requested while rendering pages on that domain. Dev and staging hosts that hold no certificate appear only in the second. For a URL it returns the most recent sandboxed scan of that host with its verdict and a link to the report, and for an IP the URLs in the public archive that resolved to that address. Results are capped at max_results attributes, 200 by default, because a busy domain can return well over a thousand subdomains. The cap and any truncation reported by the API are both surfaced as attributes rather than applied silently.
|
Thanks a lot for the module. We did some tests with some non-existing random hash values and it gives a value. Not sure if this is expected. Not really related to the module but more the service. |
|
Thanks for merging it, and for testing with random values. You were right that it was the service. The API was accepting any string as a domain and The module needed a change too: it was turning that rejection into "No ScanMalware |
New expansion + hover module: ScanMalware
Adds
misp_modules/modules/expansion/scanmalware.py, enrichingdomain,hostname,url,ip-src,ip-dstanddomain|ipattributes from the publicScanMalware API, returning
misp_standardresults built with PyMISP.No API key. The API is anonymous and a key only raises the rate limit
(600 req/min anonymous), so
moduleconfigis["api_url", "max_results"]andrequirementsis empty. The module works on a fresh install with nothing configured.Behaviour
/api/v1/ct/dns/returns subdomains seen in Certificate Transparency DNS records;/api/v1/hosts/returns hosts a browser actually resolved and requested while renderingpages on that domain. The second is what certificate-based sources structurally cannot
see: dev and staging hosts that hold no certificate. Querying only the first loses them.
link to the full report, and the IPs contacted while rendering.
max_resultsdefaults to 200. Unbounded,cloudflare.comreturns 1,207 attributes, which makes an event unusable. The cap andany truncation the API itself reports are both added as visible text attributes rather
than applied silently, so an analyst can tell a shortened list from a complete one.
scanmalware-misp/1.0User-Agent.Measured output
Tests
tests/test_scanmalware.py, 12 tests, no network access. They cover the cases that wouldotherwise degrade quietly: wildcard certificate entries (
*.example.com) being droppedrather than emitted as hosts, truncation and the result cap both being reported, a URL
being reduced to its host before lookup, an unreachable API degrading instead of raising,
and no auth header ever being sent.
Clean under the repository's own
black(line length 120, preview,string_processing),isortandflake8settings.Note on documentation: I deliberately have not committed regenerated
documentation/output. Runningdocumentation/generate.pylocally drops every modulewhose optional dependencies are missing, which produced a 4,600-line deletion across
unrelated modules. Happy to add the generated entries if you would like them in this PR,
but it seemed safer to leave that to a run with the full dependency set. The logo is
included since
moduleinforeferences it.Disclosure: I run ScanMalware, so this is a vendor-submitted module. Flagging it up
front rather than leaving it to be found in review.