database/api rework

.github/workflows/publish.yml

@@ -28,6 +28,6 @@ jobs:
           docker build . --tag ghcr.io/ngn13/massacr/scanner:latest
           docker push ghcr.io/ngn13/massacr/scanner:latest
           
-          cd ../database
-          docker build . --tag ghcr.io/ngn13/massacr/database:latest
-          docker push ghcr.io/ngn13/massacr/database:latest
+          cd ../api
+          docker build . --tag ghcr.io/ngn13/massacr/api:latest
+          docker push ghcr.io/ngn13/massacr/api:latest

.gitignore

@@ -1,3 +1,2 @@
-data/
-compose.yml
 docker-compose.yml
+compose.yml

README.md

@@ -1,64 +1,53 @@
-<h2 align="center">
-  massacr 🩸 mass IP/port scanner
-  </br>
-  a tool for scanning the entire internet
-  </br>
-  </br>
-  <img src="assets/web.png">
-</h2>
-
----
+# massacr 🩸 mass IP/port scanner toolkit
+An extensible toolkit for scanning the internet for TCP ports using SYN packets. 
+Consists of different tools and servers that interact which each other:
+```
+Scanner -> API -> Handler -> MongoDB -> Mongo-Express
+```
+- [Scanner](scanner/): SYN port scanner written in C, sends requests to the API with curl
+- [API](api/lib): Simple web API written with Flask, provides data to threaded handler
+- [Handler](api/main.py): A simple extensible Python function to process provided data. When its done processing, 
+it saves the processed data to the MongoDB database. Default handler gathers extra information about HTTP(S) servers.
+- [MongoDB](https://www.mongodb.com/what-is-mongodb): NoSQL database for storing all the data 
+- [Mongo-Express](https://github.com/mongo-express/mongo-express): Web-based MongoDB admin interface to interact with the data
 
 ## Deploy
-### Docker
-The project contains a scanner and a simple database with a web interface. Easiest way to deploy these two is to use 
-`docker-compose`. Here is an example configuration:
+Since there are multiple components of massacr, easiest 
+way to deploy is to use `docker-compose`, here is an example configuration:
 ```yml
 version: "3"
 services:
   scanner:
     image: ghcr.io/ngn13/massacr/scanner
-    command: --url=http://database:3231 --pwd=securepassword --limit=100
+    command: --url=http://api:5000 --limit=100
     depends_on:
-      - database
+      - api 
 
-  database:
-    image: ghcr.io/ngn13/massacr/database
+  api:
+    image: ghcr.io/ngn13/massacr/api
     restart: unless-stopped
     environment:
-      - PASSWORD=securepassword
-    ports:
-      - "127.0.0.1:3231:3231"
+      - API_MONGO=mongodb://mongo
+    depends_on:
+      - mongo
+
+  mongo:
+    image: mongo 
     volumes:
-      - ./data:/app/data
-```
-after deploying the containers, you can access the web interface at `http://localhost:3231`.
+      - ./db:/data/db:rw
 
-### From the source
-Another way to deploy these two applications, is to build them from the source.
-To build from source, first install all the dependencies and build tools: 
-```bash
-build-esssential libnet libnet-dev curl curl-dev go
-```
-Then clone the repository: 
-```bash
-git clone https://github.com/ngn13/massacr.git
-```
-Now change directory into the database and run the go build command:
-```bash
-cd massacr/database && go build .
-```
-Now change directory into the scanner and run the make command:
-```bash 
-cd ../scanner && make
+  interface:
+    image: mongo-express
+    depends_on:
+      - mongo
+    environment:
+      - ME_CONFIG_MONGODB_URL=mongodb://mongo
+    ports:
+      - "127.0.0.1:8081:8081"
 ```
+after deploying the containers, you can access the web interface at `http://localhost:8081`.
 
 ## Configuration
-### Database 
-All the configuration options for the database are set using environment variables:
-- `PASSWORD`: password, default is `default`
-- `PORT`: port for the web server, default is `3231`
-
 ### Scanner 
 You can list all the options with `--help`:
 ```
@@ -68,17 +57,18 @@ You can list all the options with `--help`:
 --ports    => Ports to scan for
 --limit    => Packets per second limit
 --debug    => Enable debug output
---url      => Database HTTP(S) URL
---pwd      => Database password
+--url      => API HTTP(S) URL
+--password => API password
 ```
 
-- Options are set using the `--<option>=<value>` syntax
+- Use the `--<option>=<value>` syntax to set options.
 - For the `--ports` option, you can specify a single port, or you can specify ranges with `-` (`1-100`) and multiple ports with `,` (`80,443,1234`)
 - Timeout is the time to wait after sending all the packets (in seconds), as the receiver thread may fell behind
-- `--limit` is set to 20 by default, **which is pretty slow, so you should increase it**
+- `--limit` is set to 20 by default, **which is pretty slow, so you should increase it.**
 
 > [!CAUTION]
-> Do not go overkill on the `--limit` option, you will most likely end up using all the bandwidth and crash the entire network
+> Do not go overkill on the `--limit` option, you will most likely end up using all the bandwidth 
+> and crash the entire network.
 
 Defaults for all the options are:
 ```
@@ -88,12 +78,21 @@ timeout  => 10
 ports    => common
 limit    => 20
 debug    => false
-url      => http://localhost:3231
-pwd      => default
+url      => http://localhost:5000 
+password => default
 ```
 
----
+### API/Handler
+Default API has few different options:
+- `API_MONGO`: MongoDB URL
+- `API_PASSWORD`: Password for API access
+- `API_USE_LOCAL`: Set to any value if you want API to listen only on the local interface (127.0.0.1)
+
+However you can modify these options and the handler itself by modifying [`api/main.py`](api/main.py).
+If you know a bit of python you can easily write your own handler for your specific use case. All the code 
+is well commented but feel free to create issues to ask questions if you are stuck.
 
 ## Resources 
+Here are some different resources that I used during the development:
 - [SYN scanning](https://nmap.org/book/synscan.html) (massacr does not exactly use SYN scan, it does not send RST packets)
 - [libnet](https://github.com/libnet/libnet) (provides an easy way to build and send raw network packets)

api/.gitignore

@@ -0,0 +1,4 @@
+*/__pycache__
+__pycache__
+data
+venv

api/Dockerfile

@@ -0,0 +1,19 @@
+FROM python 
+
+WORKDIR /api
+COPY lib    ./lib
+COPY *.py   ./
+COPY *.txt  ./
+
+RUN pip install -r requirements.txt
+
+ARG API_PASSWORD 
+ENV API_PASSWORD $API_PASSWORD
+
+ARG API_USE_LOCAL 
+ENV API_USE_LOCAL $API_USE_LOCAL
+
+ARG API_MONGO 
+ENV API_MONGO $API_MONGO
+
+ENTRYPOINT ["python", "/api/main.py"] 

api/lib/__init__.py

@@ -0,0 +1,99 @@
+"""
+
+ #  massacr/api | mass internet/ipv4 scanner
+ #  ========================================
+ #  written by ngn (https://ngn.tf) (2024)
+
+ #  This program is free software: you can redistribute it and/or modify
+ #  it under the terms of the GNU General Public License as published by
+ #  the Free Software Foundation, either version 3 of the License, or
+ #  (at your option) any later version.
+
+ #  This program is distributed in the hope that it will be useful,
+ #  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ #  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ #  GNU General Public License for more details.
+
+ #  You should have received a copy of the GNU General Public License
+ #  along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+"""
+
+from pymongo.mongo_client import MongoClient
+from flask import Flask, request
+from threading import Thread 
+import logging as log
+
+class API:
+    def __init__(self, mongo_url: str, password: str, host: str, port: int) -> None:
+        self.handler = None
+        self.app = Flask(__name__)
+        self.mongo = MongoClient(mongo_url)
+        self.password = password
+        self.host = host 
+        self.port = port
+
+        werkzeug_log = log.getLogger("werkzeug")
+        werkzeug_log.setLevel(log.ERROR)
+        log.basicConfig(level=log.INFO, format="%(asctime)s [%(levelname)s]: %(message)s")
+
+        self.app.add_url_rule("/scanner/check", "/scanner/check", self.route_check)
+        self.app.add_url_rule("/scanner/add", "/scanner/add", self.route_add)
+
+    def db_add(self, ip: str, port: int, extra):
+        db = self.mongo["results"]
+        col = db["hosts"]
+        data = {
+            "ipv4": ip,
+            "ports": [port],
+            "data": [extra] if len(extra.keys()) != 0 else [],
+        }
+
+        found = col.find_one({"ipv4":  ip})
+        if not found:
+            return col.insert_one(data)
+
+        if not port in found["ports"]:
+            col.update_one({"ipv4": ip}, {"$push": {"ports": port}})
+
+        if len(extra.keys()) != 0:
+            col.update_one({"ipv4": ip}, {"$push": {"data": extra}})
+
+    def set_handler(self, func):
+        self.handler = func
+
+    def run(self):
+        log.info("Trying to connect to mongodb")
+        try:
+            log.info(f"Connected to mongodb version {self.mongo.server_info()['version']}")
+        except Exception as e:
+            log.critical(f"Database connection failed: {e}")
+            exit(1)
+
+        log.info(f"Starting API server on {self.host}:{self.port}")
+        self.app.run(host=self.host, port=self.port)
+
+    def route_check(self):
+        password = request.args["pass"]
+        if self.password != password:
+            return "Bad password", 403
+
+        log.info(f"Password check successful for {request.remote_addr}")
+        return "OK", 200
+
+    def route_add(self):
+        password = request.args["pass"]
+        port = request.args["port"]
+        ip = request.args["ip"]
+
+        if self.password != password:
+            return "Bad password", 403
+
+        if self.handler == None:
+            self.db_add(ip, int(port), {})
+
+        t = Thread(target=self.handler, args=(ip,int(port),))
+        t.daemon = True
+        t.run()
+
+        return "OK", 200

api/main.py

@@ -0,0 +1,53 @@
+from os import environ
+import requests as req
+from lib import *
+
+def http_handler(ip, port):
+    # if not an HTTP(S) server skip
+    if port != 80 and port != 443:
+        return api.db_add(ip, port, {})
+
+    # if its a possible HTTP(S) server send a request 
+    # and gather simple data
+    protocol = "http" if (port == 80) else "https" 
+    try:
+        res = req.get(f"{protocol}://{ip}", verify=False)
+    except:
+        return api.db_add(ip, port, {})
+
+    server = res.headers["Server"]
+    body = res.text
+
+    # api.db_add adds ip and port data to 'results' database,
+    # if ip already exists, it just appends the port 
+    # it also adds any extra information you provide with 
+    # the last argument
+    api.db_add(ip, port, {
+        "http_server": server,
+        "http_body": body,
+    })
+    # instead of 'api.db_add' you can access 
+    # the mongo driver with 'api.mongo' and directly modify
+    # the database however you like
+
+if __name__ == "__main__":
+    mongo_url = environ["API_MONGO"] if "API_MONGO" in environ else "mongodb://localhost"
+    password  = environ["API_PASSWORD"] if "API_PASSWORD" in environ else "default"
+    host      = "127.0.0.1" if "API_USE_LOCAL" in environ else "0.0.0.0"
+    port      = 5000
+
+    # creating the API
+    api = API(
+        mongo_url, # MongoDB database URL
+        password,  # API password 
+        host,      # Address to listen on
+        port,      # Port to listen on
+    )
+
+    # use our custom HTTP handler
+    # if no hanlder is set, API will just store IP and port info 
+    # on the 'results' database
+    api.set_handler(http_handler)
+
+    # start the API
+    api.run()

api/requirements.txt

@@ -0,0 +1,3 @@
+Flask
+pymongo
+requests