Skip to content

Latest commit

 

History

History
290 lines (247 loc) · 24.8 KB

Usage.md

File metadata and controls

290 lines (247 loc) · 24.8 KB
Raw

Usage

Client

Intel Owl main objective is to provide a single API interface to query in order to retrieve threat intelligence at scale.

There are multiple ways to interact with the Intel Owl APIs,

  1. IntelOwl-ng (Web Interface)

    • Inbuilt Web interface with dashboard, visualizations of analysis data, easy to use forms for requesting new analysis, tags management and more features
    • Live Demo
    • Built with Angular 10+ and available on GitHub.

  2. pyIntelOwl (CLI/SDK)

    • Official client that is available at: PyIntelOwl,
    • Can be used as a library for your own python projects or...
    • directly via the command line interface.

Hint: Tokens Creation

The server authentication is managed by API tokens. So, if you want to interact with Intel Owl, you have to create one or more unprivileged users from the Django Admin Interface and then generate a token for those users. Afterwards you can leverage the created tokens with the Intel Owl Client.

Analyzers customization

You can create new analyzers based on already existing modules by changing the configuration values inside configuration/analyzer_config.json. This file is mounted as a docker volume, so you won't need to rebuild the image.

You may want to change this configuration to add new analyzers or to change the configuration of some of them. The name of the analyzers can be changed at every moment based on your wishes.

The following are all the keys that you can change without touching the source code:

  • disabled: you can choose to disable certain analyzers, then they won't appear in the dropdown list and won't run if requested.
  • leaks_info: if set, in the case you specify via the API that a resource is sensitive, the specific analyzer won't be executed
  • external_service: if set, in the case you specify via the API to exclude external services, the specific analyzer won't be executed
  • supported_filetypes: can be populated as a list. If set, if you ask to analyze a file with a different mimetype from the ones you specified, it won't be executed
  • not_supported_filetypes: can be populated as a list. If set, if you ask to analyze a file with a mimetype from the ones you specified, it won't be executed
  • observable_supported: can be populated as a list. If set, if you ask to analyze an observable that is not in this list, it won't be executed. Valid values are: ip, domain, url, hash, generic.
  • soft_time_limit: this is the maximum time (in seconds) of execution for an analyzer. Once reached, the task will be killed (or managed in the code by a custom Exception). Default 300.
  • queue: this takes effects only when multi-queue is enabled. Choose which celery worker would execute the task: local (ideal for tasks that leverage local applications like Yara), long (ideal for long tasks) or default (ideal for simple webAPI-based analyzers).

Hint: Advanced Configuration

You can also modify analyzer specific parameters from the configuration file or even at the time of requesting an analysis. See Customize analyzer execution at time of request
Example: add an analyzer configuration for your own Yara signatures
    "Yara_Scan_Custom_Signatures": {
        "type": "file",
        "python_module": "yara.Yara",
        "description": "Executes Yara with custom signatures",
        "config": {
            "queue": "default",
            "soft_time_limit": 100,
        },
        "params": {
            "directories_with_rules": ["/opt/deploy/yara/custom_signatures"]
        }
    }

Connectors customization

Connectors being optional are disabled by default. You can enable them by changing the configuration values inside configuration/connector_config.json. This file is mounted as a docker volume, so you won't need to rebuild the image.

The following are all the keys that you can change without touching the source code:

  • disabled: similar to analyzers
  • soft_time_limit: similar to analyzers
  • queue: similar to analyzers
  • maximum_tlp (default WHITE, choices WHITE, GREEN, AMBER, RED): specify with the maximum TLP of the analysis upto which the connector is allowed to run. (e.g. if maximum_tlp is GREEN, it would run for analysis with TLPs WHITE and GREEN). To learn more about TLPs see TLP Support.

Warning

Changing other keys can break an analyzer or connector. In that case, you should think about duplicating the configuration entry or python module with your changes.

Managing Analyzers and Connectors

All plugins i.e. analyzers and connectors have kill and retry actions. In addition to that, all docker-based analyzers and connectors have a healthcheck action to check if their associated instances are up or not.

  • kill:

    To stop a plugin whose status is running/pending:

    • GUI: Buttons on reports table on job result page.
    • PyIntelOwl: IntelOwl.kill_analyzer and IntelOwl.kill_connector function.
    • CLI: $ pyintelowl jobs kill-analyzer <job_id> <analyzer_name> and $ pyintelowl jobs kill-connector <job_id> <connector_name>
    • API: PATCH /api/job/{job_id}/analyzer/{analyzer_name}/kill and PATCH /api/job/{job_id}/connector/{connector_name}/kill

  • retry:

    To retry a plugin whose status is failed/killed:

    • GUI: Buttons on reports table on job result page.
    • PyIntelOwl: IntelOwl.retry_analyzer and IntelOwl.retry_connector function,
    • CLI: $ pyintelowl jobs retry-analyzer <job_id> <analyzer_name> and $ pyintelowl jobs retry-connector <job_id> <connector_name>
    • API: PATCH /api/job/{job_id}/analyzer/{analyzer_name}/retry and PATCH /api/job/{job_id}/connector/{connector_name}/retry

  • healthcheck:

    To check if docker container or external platform associated with an analyzer or connector respectively are up or not:

    • GUI: Buttons on analyzers table and connectors table.
    • PyIntelOwl: IntelOwl.analyzer_healthcheck and IntelOwl.connector_healthcheck methods.
    • CLI: $ pyintelowl analyzer-healthcheck <analyzer_name> and $ pyintelowl connector-healthcheck <connector_name>
    • API: GET /api/analyzer/{analyzer_name}/healthcheck and GET /api /connector/{connector_name}/healthcheck

TLP Support

IntelOwl supports the Traffic Light Protocol (TLP) to facilitate sharing of job analysis results.

Following are the indicators available when requesting an analysis (in the order of increasing sharing restrictions):

  1. WHITE: no restriction
  2. GREEN: disable analyzers that could impact privacy
  3. AMBER: disable analyzers that could impact privacy and limit view permissions to my group
  4. RED: disable analyzers that could impact privacy, limit view permissions to my group and do not use any external service

These indicators when used with maximum_tlp (option available in connectors), give you the control of what information is shared to the external platforms.

Available Analyzers

Analyzers list

The following is the list of the available analyzers you can run out-of-the-box. You can also navigate the same list via the,

File analyzers:
  • File_Info: static generic File analysis (hashes, magic and exiftool)
  • PDF_Info: static PDF analysis (peepdf + pdfid)
  • Rtf_Info: static RTF analysis (Oletools)
  • Doc_Info: static generic document analysis (Oletools)
  • Xlm_Macro_Deobfuscator: XlmMacroDeobfuscator deobfuscate xlm macros
  • Doc_Info_Experimental: static document analysis with new features to analyze XLM macros, encrypted macros and more (combination of Oletools and XLMMacroDeobfuscator)
  • PE_Info: static PE analysis with pefile
  • Signature_Info: PE signature extractor with osslsigncode
  • Speakeasy: FireEye Speakeasy binary emulation
  • Strings_Info_Classic: strings extraction
  • Strings_Info_ML: strings extraction plus strings ranking based on Machine Learning. Leverages Stringsifter
  • VirusTotal_v3_Get_File_And_Scan: check file hash on VirusTotal. If not already available, send the sample and perform a scan
  • VirusTotal_v3_Get_File: check only the file hash on VirusTotal (this analyzer is disabled by default to avoid multiple unwanted queries. You have to change that flag in the config to use it)
  • VirusTotal_v2_Get_File: check file hash on VirusTotal using old API endpoints (this analyzer is disabled by default. You have to change that flag in the config to use it)
  • VirusTotal_v2_Scan_File: scan a file on VirusTotal using old API endpoints (this analyzer is disabled by default. You have to change that flag in the config to use it)
  • Intezer Scan: scan a file on Intezer. Register for a free community account here
  • Cuckoo_Scan: scan a file on Cuckoo (this analyzer is disabled by default. You have to change that flag in the config to use it)
  • HybridAnalysis_Get_File: check file hash on HybridAnalysis sandbox reports
  • OTX_Check_Hash: check file hash on Alienvault OTX
  • MISP_Check_Hash: check a file hash on a MISP instance
  • MISPFIRST_Check_Hash: check a file hash on the FIRST MISP instance
  • Yara_Scan_ATM_MALWARE: scan a file with the ATM malware yara rules
  • Yara_Scan_Community: scan a file with the community yara rules
  • Yara_Scan_Dail_Ioc: scan a file with StrangerealIntel Daily IOC yara rules
  • Yara_Scan_Florian: scan a file with Neo23x0 yara rules
  • Yara_Scan_Intezer: scan a file with Intezer yara rules
  • Yara_Scan_Inquest: scan a file with Inquest yara rules
  • Yara_Scan_McAfee: scan a file with McAfee yara rules
  • Yara_Scan_Samir: scan a file with Samir Threat Hunting yara rules
  • Yara_Scan_Stratosphere: scan a file with Stratosphere yara rules
  • Yara_Scan_FireEye: scan a file with FireEye yara rules
  • Yara_Scan_ReversingLabs: scan a file with ReversingLabs yara rules
  • Yara_Scan_Custom_Signatures: scan a file with your own added signatures
  • MalwareBazaar_Get_File: Check if a particular malware sample is known to MalwareBazaar
  • PEframe_Scan: Perform static analysis on Portable Executable malware and malicious MS Office documents with PeFrame
  • Cymru_Hash_Registry_Get_File: Check if a particular file is known to be malware by Team Cymru
  • Thug_HTML_Info: Perform hybrid dynamic/static analysis on a HTML file using Thug low-interaction honeyclient
  • Capa_Info: Capa detects capabilities in executable files
  • BoxJS_Scan_Javascript: Box-JS is a tool for studying JavaScript malware.
  • APKiD_Scan_APK_DEX_JAR: APKiD identifies many compilers, packers, obfuscators, and other weird stuff from an APK or DEX file.
  • Quark_Engine_APK: Quark Engine is an Obfuscation-Neglect Android Malware Scoring System.
  • IntelX_Phonebook: IntelligenceX is a search engine and data archive. Fetches emails, urls, domains associated with an observable.
  • UnpacMe_EXE_Unpacker: UnpacMe is an automated malware unpacking service
  • Triage_Scan: leverage Triage sandbox environment to scan various files
  • Manalyze: Manalyze performs static analysis on PE executables to detect undesirable behavior.
  • MWDB_Scan: mwdblib Retrieve malware file analysis from repository maintained by CERT Polska MWDB.
  • Qiling: Qiling qiling binary emulation.
  • Malpedia_Scan: scan a binary or a zip file (pwd:infected) against all the yara rules available in Malpedia
  • HashLookupServer_Get_File: check if a md5 or sha1 is available in the database of known file hosted by CIRCL
  • FileScan_Upload_File: Upload your file to extract IoCs from executable files, documents and scripts via FileScan.io API.
  • Dragonfly_Emulation: Emulate malware against Dragonfly sandbox by Certego S.R.L.
Observable analyzers (ip, domain, url, hash)
  • VirusTotal_v3_Get_Observable: search an observable in the VirusTotal DB
  • VirusTotal_v2_Get_Observable: search an observable in the VirusTotal DB using the old API endpoints (this analyzer is disabled by default. You have to change that flag in the config to use it)
  • HybridAnalysis_Get_Observable: search an observable in the HybridAnalysis sandbox reports
  • OTXQuery: scan an observable on Alienvault OTX
  • TalosReputation: check an IP reputation from Talos
  • Stratosphere_Blacklist: Cross-reference an IP from blacklists maintained by Stratosphere Labs
  • BitcoinAbuse : Check a BTC address against bitcoinabuse.com, a public database of BTC addresses used by hackers and criminals.
  • Robtex_Forward_PDNS_Query: scan a domain against the Robtex Passive DNS DB
  • Robtex_Reverse_PDNS_Query: scan an IP against the Robtex Passive DNS DB
  • Robtex_IP_Query: get IP info from Robtex
  • GoogleSafebrowsing: Scan an observable against GoogleSafeBrowsing DB
  • GoogleWebRisk: Scan an observable against WebRisk API (Commercial version of Google Safe Browsing). Check the docs to enable this properly
  • GreyNoiseCommunity: scan an IP against the Community Greynoise API (no API key required)
  • GreyNoise: scan an IP against the Greynoise API (requires API key)
  • CIRCLPassiveDNS: scan an observable against the CIRCL Passive DNS DB
  • CIRCLPassiveSSL: scan an observable against the CIRCL Passive SSL DB
  • MaxMindGeoIP: extract GeoIP info for an observable
  • AbuseIPDB: check if an ip was reported on AbuseIPDB
  • Fortiguard: scan an observable with the Fortiguard URL Analyzer
  • TorProject: check if an IP is a Tor Exit Node
  • MISP: scan an observable on a MISP instance
  • MISPFIRST: scan an observable on the FIRST MISP instance
  • DNSDB: scan an observable against the Passive DNS Farsight Database (support both v1 and v2 versions)
  • Shodan_Search: scan an IP against Shodan Search API
  • Shodan_Honeyscore: scan an IP against Shodan Honeyscore API
  • HoneyDB_Get: HoneyDB IP lookup service
  • HoneyDB_Scan_Twitter: scan an IP against HoneyDB.io's Twitter Threat Feed
  • Hunter: Scans a domain name and returns set of data about the organisation, the email address found and additional information about the people owning those email addresses.
  • Censys_Search: scan an IP address against Censys View API
  • MalwareBazaar_Get_Observable: Check if a particular malware hash is known to MalwareBazaar
  • MalwareBazaar_Google_Observable: Check if a particular IP, domain or url is known to MalwareBazaar using google search
  • ONYPHE: search an observable in ONYPHE
  • Threatminer_PDNS: retrieve PDNS data from Threatminer API
  • Threatminer_Reports_Tagging: retrieve reports from Threatminer API
  • Threatminer_Subdomains: retrieve subdomains from Threatminer API
  • URLhaus: Query a domain or URL against URLhaus API.
  • Google_DNS: Retrieve current domain resolution with Google DoH (DNS over HTTPS)
  • CloudFlare_DNS: Retrieve current domain resolution with CloudFlare DoH (DNS over HTTPS)
  • CloudFlare_Malicious_Detector: Leverages CloudFlare DoH to check if a domain is related to malware
  • Classic_DNS: Retrieve current domain resolution with default DNS
  • Auth0: scan an IP against the Auth0 API
  • Securitytrails_IP_Neighbours: scan an IP against Securitytrails API for neighbour IPs
  • Securitytrails_Details: scan a domain against Securitytrails API for general details
  • Securitytrails_Subdomains: scan a domain against Securitytrails API for subdomains
  • Securitytrails_Tags: scan a domain against Securitytrails API for tags
  • Securitytrails_History_WHOIS: scan a domain against Securitytrails API for historical WHOIS
  • Securitytrails_History_DNS: scan a domain against Securitytrails API for historical DNS
  • Cymru_Hash_Registry_Get_Observable: Check if a particular hash is available in the malware hash registry of Team Cymru
  • Tranco: Check if a domain is in the latest Tranco ranking top sites list
  • Thug_URL_Info: Perform hybrid dynamic/static analysis on a URL using Thug low-interaction honeyclient
  • Pulsedive_Active_IOC: Scan indicators and retrieve results from Pulsedive's API.
  • CheckDMARC: An SPF and DMARC DNS records validator for domains.
  • Whoisxmlapi: Fetch WHOIS record data, of a domain name, an IP address, or an email address.
  • WhoIs_RipeDB_Search : Fetch whois record data of an IP address from Ripe DB using their search API (no API key required)
  • UrlScan_Search: Search an IP/domain/url/hash against URLScan API
  • UrlScan_Submit_Result: Submit & retrieve result of an URL against URLScan API
  • Mnemonic_PassiveDNS : Look up a domain or IP using the Mnemonic PassiveDNS public API.
  • Phishtank: Search an url against Phishtank API
  • Phishstats: Search PhishStats API to determine if an IP/URL/domain is malicious.
  • Quad9_DNS: Retrieve current domain resolution with Quad9 DoH (DNS over HTTPS)
  • Quad9_Malicious_Detector: Leverages Quad9 DoH to check if a domain is related to malware
  • DNStwist: Scan a url/domain to find potentially malicious permutations via dns fuzzing. dnstwist repo
  • IPInfo: Location Information about an IP
  • Zoomeye: Zoomeye Cyberspace Search Engine recording information of devices, websites, services and components etc..
  • Triage_Search: Search for reports of observables or upload from URL on triage cloud
  • InQuest_IOCdb: Indicators of Compromise Database by InQuest Labs
  • InQuest_REPdb: Search in InQuest Lab's Reputation Database
  • InQuest_DFI: Deep File Inspection by InQuest Labs
  • XForceExchange: scan an observable on IBM X-Force Exchange
  • Renderton: get screenshot of a web page using rendertron (puppeteer) renderton repo
  • SSAPINet: get a screenshot of a web page using screenshotapi.net (external source); additional config options can be added to extra_api_params in the config.
  • FireHol_IPList: check if an IP is in FireHol's IPList
  • ThreatFox: search for an IOC in ThreatFox's database
  • OpenCTI: scan an observable on an OpenCTI instance
  • Intezer_Get: check if an analysis related to a hash is available in Intezer. Register for a free community account here.
  • MWDB_Get: mwdblib Retrieve malware file analysis by hash from repository maintained by CERT Polska MWDB.
  • YETI (Your Everyday Threat Intelligence): scan an observable on a YETI instance.
  • HashLookupServer_Get_Observable: check if a md5 or sha1 is available in the database of known file hosted by CIRCL
  • ClamAV: scan a file via the ClamAV AntiVirus Engine
  • Spyse: Scan domains, IPs, emails and CVEs using Spyse's API. Register here.
  • FileScan_Search: Finds reports and uploaded files by various tokens, like hash, filename, verdict, IOCs etc via FileScan.io API.
Generic analyzers (email, phone number, etc.; anything really)

Some analyzers require details other than just IP, URL, Domain, etc. We classified them as generic Analyzers. Since the type of field is not known, there is a format for strings to be followed.

  • EmailRep: search an email address on emailrep.io
  • WiGLE: Maps and database of 802.11 wireless networks, with statistics, submitted by wardrivers, netstumblers, and net huggers.
  • CRXcavator: scans a chrome extension against crxcavator.io
  • Darksearch_Query: Search a keyword against darksearch.io's search API. It's possible to make complex queries using boolean logic. For example, OSINT AND CTI OR intelowl NOT hack is a valid observable name.
  • Dehashed_Search: Query any observable/keyword against https://dehashed.com's search API.
  • CryptoScamDB_CheckAPI: Scan a cryptocurrency address, IP address, domain or ENS name against the CryptoScamDB API.
Extra analyzers

Additional analyzers that can be enabled per your wish.

Available Connectors

Connectors are designed to run after every successful analysis which makes them suitable for automated threat-sharing. They support integration with other SIEM/SOAR projects, specifically aimed at Threat Sharing Platforms.

Connectors list

The following is the list of the available connectors. You can also navigate the same list via the,

  • MISP: automatically creates an event on your MISP instance, linking the successful analysis on IntelOwl.
  • OpenCTI: automatically creates an observable and a linked report on your OpenCTI instance, linking the the successful analysis on IntelOwl.
  • YETI: YETI = Your Everyday Threat Intelligence. find or create observable on YETI, linking the successful analysis on IntelOwl.

To contribute to the project, see Contribute.