Add filesystem source

[rshipp]

Point it at a file/directory and it will recursively read and extract artifacts similar to iocextract CLI.

[deadbits]

Maybe consider using watchdog to monitor for changes to that directory automatically and extract artifacts from any newly added files:

• https://pypi.org/project/watchdog/

[rshipp]

Interesting. I'm not sure if this will work here, I need something that doesn't run continuously and just gives me a flag or something I can check to see if there have been changes (mtime maybe?). I haven't thought too far into it.

[deadbits]

I personally think continuous monitoring (or even periodically per day/hour) of a path could be great.

My use case premise was something like: Analysts put APT reports in a given directory or share drive, Threat Ingestors uses watchdog (or similar) to watch path, analyst of T.I. adds that directory as a source and all newly added reports have their artifacts automatically extracted.

Potentially on the user's pipeline they could just feed the results into an alerting list for IPS, hashes, or add YARA sigs to their internal ruleset application.

[rshipp]

The problem with continuous is just that ThreatIngestor isn't a persistent process, it's meant to run quickly once every X minutes / hours.

One workaround might be creating an SQS worker that uses watchdog, and adds a job to the queue if it notices a change. Then ThreatIngestor can just watch the queue and run on the files when it sees a new job.

[deadbits]

Ahh gotcha 👍 i misunderstood

[rshipp]

This is now possible due to #52. @needmorecowbell has a SQS worker using watchdog locally. Need to add docs on SQS workflow (#17) and figure out how we're going to publish the "SQS worker" concept/examples... maybe wait until #40 is done too so it's not tied to SQS specifically.

[rshipp]

Closed by #60. See https://threatingestor.readthedocs.io/en/latest/extras.html#fswatcher for docs.