Informant is a web-based contactless 'active' reconnaissance asset discovery and vulnerability assessment platform. Utilising scan data from various Internet-wide scanning and passive DNS projects to discover and evaluate the security of a given exposed Internet-facing attack surface. All known-vulnerability data is pulled from the National Vulnerability Database (NVD) which the U.S. government maintains.
The Informant platform has been developed as part of my final year degree Hons project. However, its potential use cases strive far beyond academia, with functionality rivalling that of enterprise solutions. Key features include: Historical data viewing, Known-vulnerability assessment, maximum total rDDoS potential, risky port analysis, automated statistical data overviews and vulnerable by default configuration common vulnerabilities and exposures (CVE) detection. This tool's indirect detection methods used to retrieve asset information ensure the targeted assets are never aware of your reconnaissance attempts.
Please be aware that the project is currently in early development and should be treated as such, with the experimental vulnerable by default configuration flagging feature being in its infancy. This project was intended to be an academic proof of concept, which is reflected in the quality of the code. However, large sections of code are currently being re-written to improve the overall readability and performance of the codebase.
Research surrounding the tool can be found here.
Informant is currently under heavy development and only comes as a standalone web-based application. The application relies on numerous dependencies and requires access to at least one valid Shodan, Censys or BinaryEdge API key to function. In addition to a Redis and MongoDB instance. All input and retrieved data is stored locally.
- Python 3.6 or later
- MongoDB 2.2 or later
- Redis Server
- Pip3
- Flask
- Flask-PyMongo
- Censys
- Shodan
- Pybinaryedge
- Dnsdb
- Tldextract
- RapidFuzz
- PyMongo
- Requests
- Redis
- RQ (Redis Queue)
- Beautifulsoup4
- WTForms
- Urllib3
- Python-dotenv
- More-itertools
- Bson
- Collections
- Netaddr
- Ipaddress
- Ipwhois
- Pyvis
*Python dependencies will shrink as the tool evolves and the codebase is further optimised.
- Install and start MongoDB
# Install MongoDB
sudo apt-get install -y mongodb
# Start MongoDB service
sudo systemctl start mongodb- Install cve-search and populate MongoDB (timely process . . .)
note cve-search is designed to work on Linux only - However can be adapted for Windows, get in touch if you need a hand. This entire step can be skipped however the tool will not be able to conduct the known vulnerability identification process.
# Download repo
sudo git clone https://github.com/cve-search/cve-search.git
# Install dependencies
sudo pip3 install -r requirements.txt
# Create and populate CVEDB in MongoDB
./sbin/db_mgmt_cpe_dictionary.py -p
# then . . patience . .
./sbin/db_mgmt_json.py -p
# then . . . a lot more patience . . .
./sbin/db_updater.py -c- Install and start redis server
# Install Redis Server
sudo apt install redis-server
# Start the Redis Server
sudo systemctl start redis- Install Informant
# Download repo
sudo git clone https://github.com/LA-Shill/Informant.git
# Access directory
cd Informant
# Install dependencies
sudo pip3 install -r requirements.txt- Configure the following parameters within the default .env file according to your setup (in most instances the configure below will work straight out of the box but is highly insecure)
DB_HOST= 127.0.0.1
DB_PORT= 27017
CORE_MONGO_DB= "mongodb://127.0.0.1:27017/core"
VUL_MONGO_DB= "mongodb://127.0.0.1:27017/cvedb"
REDISTOGO_URL= "redis://:@127.0.0.1:6379/dev"-
Informant is currently programmatically hard limited to retrieving a maximum of 1000 IWS records per target range to help prevent against accidental excessive credit usage.
-
Performance deteriorates significantly when handling multiple millions of records. This is being worked on as a priority, with multiple solutions already in the works.
-
The active based fraudulent geolocation feature is currently disabled and will be re-enabled in the next major patch. All operational methods within Informant follow the CAR process and do not directly connect with target networks.
-
Percentage change statistics are currently disabled. This functionality will be re-enabled during the next major code overhaul due to performance issues.
-
Censys and Onyphe are not currently utilised to their full potential. Data retrieval from Censys is limited to a finite array of protocols to lessen the platforms' credit use during large scans - this will be addressed in the next major update.
- Create a worker (preferably run this in the background using screen)
# Start RQ worker (redis)
# Please note that the REDISTOGO_URL must be set in memory! Alternatively replace line #13 in the worker.py script with your correct connection details.
python3 worker.py- Start the development server
# Start dev web server on local address(s) (port is modifiable within the wsgi.py file and a app.ini file is provided for use with third-party web servers such as Nginx)
python3 wsgi.py- Navigate to INFORMANT in your browser of choice at: 127.0.0.1:5000
- Finally, add your API key(s) and configure your parameters by navigating to the settings tab located at: 127.0.0.1:5000/settings
- Scan your first IP or CIDR block by utilising the primary search bar or by creating a new project and manual initialising an asset scan.
-
The latest data within the time frame selected will be displayed.
-
Data displayed is in time-series and linked directly with the interactive timeline bar.
-
Overview of Passive DNS records flagged as external, for asset detection outwith your targeted network ASN.
-
Overview of risky IWS assets, please note that devices flagged as obfuscated services are not deemed a risk.
-
Data drilldowns and exportation is available via the dropdown menu and hyperlinked IP addresses. Please note that no hyperlink within the tool will connect you to the target system.
Current list of program flags
- Breakdown of all ingested Internet-wide scanning data.
- Data exportable in CSV format.
- Breakdown of all passive DNS data.
- Data exportable in CSV format.
- Interactive graph showing the exposed attack surface, based on current timeline selection and constructed using IWS and PDNS data.
- Data exportable in HTML format.
- The latest data within the time frame selected will be displayed.
- Overview of all data stored on an asset.
- Powerful non-direct asset discovery capabilities
- Passive DNS hostname enrichment
- Fraudulent geolocation validation (direct scanning feature, currently disabled)
- Vulnerable by default CVE identification
- Service banner to common platform enumeration (CPE) reconstruction
- Known-vulnerability risk assessment
- Maximum asset bandwidth amplification factor (BAF) rDDoS prediction
- Obfuscated service banner detection
- High-risk port list
| Name | API Key Required |
|---|---|
| Shodan | Yes |
| *Censys | Yes |
| BinaryEdge | Yes |
| **Onyphe | Yes |
*Data ingest is limited to popular protocols/ports (Limitation due to free academic requirement of the project and will be rectified in the future)
** Platform needs to be used in conjunction with another Internet-wide Scanner (Limitation due to free academic requirement of the project and will be rectified in the future)
| Name | API Key Required |
|---|---|
| ThreatCrowd | No |
| ThreatMiner | No |
| Robtex | No |
| Daloo | No |
| FarSight | Yes |
| DNSGrep | No |
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
- Major code clean up (Almost a complete rewrite!! This project was solely academic at first hence the 'scripty' nature of certain code segments)
- Tidy up and reintegrate unit tests
- Improve PDNS enrichment I/O query performance (on datasets > 250,000 records)
- Automate NVD database updating
- Implement automated task scheduling
- Complete user validation
- Implement GreyNoise support
- Integrate the concept of workspaces, similar to Kibana
- Dockerize project
Open to additional feature requests.
Please note I am by NO means a software developer, so feel free to suggest improvements and changes! 😊