Skip to content

Commit

Permalink
Initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
Nicolas THIBAUT committed Jun 26, 2018
0 parents commit d64fc9c
Show file tree
Hide file tree
Showing 34 changed files with 1,428 additions and 0 deletions.
2 changes: 2 additions & 0 deletions .gitignore
@@ -0,0 +1,2 @@
*~
venv
80 changes: 80 additions & 0 deletions README.md
@@ -0,0 +1,80 @@
## *UPPERSAFE Open Source Firewall*

OSFW is a firewall, fully written in Python 3, that provides an IP / domain filtering based on a collection of public threat intelligence feeds.
It blocks in real time *incoming* / *outcoming* traffic considered as *malicious* (matching the filtering rules automatically set up for each threat).
It also provides a secure DNS service that blocks different kind of cyber threats (phishing websites, malware hosting, C&C servers, etc).

## Components

OSFW includes 3 main components:

- **osfw-sensor** (in charge of monitoring and logging the requests blocked by the firewall)
- **osfw-syncfw** (in charge of collecting and syncing the threat intelligence feeds)
- **osfw-webapp** (in charge of offering a management dashboard)

## Quick start

Setup the virtual environment:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Start the firewall components:

bash run.sh

Attach a screen:

screen -r osfw-sensor
screen -r osfw-syncfw
screen -r osfw-webapp

## Configuration

To enable the secure DNS service, simply create a symbolic link of the *firewall.conf* file to the unbound configuration directory with the following command:

ln -s "$PWD/assets/unbound.conf" /etc/unbound/unbound.conf.d/firewall.conf

It is possible to customize the behaviour of the firewall by editing the default *config.yaml* file.

One of the reasons you would want to edit this file is to unblock specific websites.
It happens that some legit and top ranked websites got blocked because of different purposes, most of the time one of the following:

- Their users can upload files on the main domain (file transfer providers or cloud storage providers)
- Their users can upload files or even web pages on a subdomain (hosting providers)
- Their users can perform URL redirect (link shortener websites)

To prevent these websites from being blocked, you can specify them as a list in the configuration file.
In case you want to edit the default list, you can use a magic keyword ".tld" that will match any top level domain and the following second level domain names:

- .co.tld
- .com.tld
- .net.tld
- .org.tld
- .edu.tld
- .gov.tld

There is also a way to make a rule act as a subdomain wildcard, to do so you need to start the rule with a "." such as the ones already in the configuration file.

## Dependencies

- python3
- iptables
- unbound
- screen

## Python requirements

- pyyaml
- colorlog
- requests
- flask
- sqlalchemy
- pebble

## Support

Nicolas THIBAUT (nicolas[@]uppersafe[.]com)

https://www.patreon.com/dev2lead/memberships
Empty file added assets/logs.txt
Empty file.
Binary file added assets/sqlite.db
Binary file not shown.
171 changes: 171 additions & 0 deletions config.yaml
@@ -0,0 +1,171 @@
---
db: assets/sqlite.db
debug: yes
threads: 32
flaskHost: 127.0.0.1
flaskPort: 8000
userAgent: UPPERSAFE (Open Source Firewall Project) https://github.com/dev2lead/uppersafe-osfw
queryTimeout: 10
refreshDelay: 3600
filterMode: classic
groupRange: yes
monitor: /var/log/messages
network:
eth: eth+
ppp: ppp+
tun: tun+
unbound:
verbosity: 1
hide-version: yes
interface: 0.0.0.0
access-control: 10.0.0.0/8 allow
feeds:
- alienvault
- blocklist
- ciarmy
- greensnow
- openphish
- phishtank
- ransomware
- spamhaus
# - uppersafe
exemptions:
# - 217.78.11.148
- 127.0.0.1
- .163.tld
- .360.tld
- .abuse.co
- adf.ly
- .adjust.tld
- .akamai.tld
- .akamaiedge.tld
- .akamaihd.tld
- .alibaba.tld
- .alienvault.tld
- .aliexpress.tld
- .amazon.tld
- amzn.to
- .aol.tld
- .aolcdn.tld
- .aryaka.tld
- .baidu.tld
- .bing.tld
- bit.ly
- .bitly.tld
- .bitbucket.tld
- .blocklist.tld
- .box.tld
- buff.ly
- .buffer.tld
- .cinsscore.tld
- .cloudflare.tld
- .discord.tld
- .discordapp.tld
- .doubleclick.tld
- .dropbox.tld
- .duckduckgo.tld
- .dyn.tld
- .dyndns.tld
- .emailretargeting.tld
- .facebook.tld
- .fb.tld
- .fbcdn.tld
- .free.tld
- .freecdn.tld
- g.co
- .gandi.tld
- .ggpht.tld
- .github.tld
- .githubusercontent.tld
- .gitlab.tld
- .gmail.tld
- goo.gl
- .google.tld
- .googleadservices.tld
- .googleadwords.tld
- .googleapis.tld
- .googleapps.tld
- .googlecache.tld
- .googlecode.tld
- .googledocs.tld
- .googledrive.tld
- .googlegroups.tld
- .googleimages.tld
- .googlemail.tld
- .googleplus.tld
- .googleproxy.tld
- .googlesearch.tld
- .googlesyndication.tld
- .googletranslate.tld
- .googleusercontent.tld
- .greensnow.tld
- .gstatic.tld
- .gsuite.tld
- .hotmail.tld
- .href.tld
- .icloud.tld
- .iconfinder.tld
- .imageshack.tld
- .imgur.tld
- .instagram.tld
- .line.tld
- .linkedin.tld
- .live.tld
- .lnkd.tld
- .mailchimp.tld
- .mailgun.tld
- .mailjet.tld
- .mandrill.tld
- .mandrillapp.tld
- .maxcdn.tld
- .messenger.tld
- .microsoft.tld
- .microsoftonline.tld
- .microsoftstore.tld
- .msdn.tld
- .msft.tld
- .msftncsi.tld
- .naver.tld
- .netflix.tld
- .office.tld
- .office365.tld
- .openphish.tld
- .outlook.tld
- .ovh.tld
- ow.ly
- .parastorage.tld
- .phishtank.tld
- .pinterest.tld
- .postimage.tld
- .postimg.tld
- .prestashop.tld
- .qq.tld
- .quora.tld
- .reddit.tld
- .shopify.tld
- .similarcdn.tld
- .similarweb.tld
- .sina.tld
- .sohu.tld
- .spamhaus.tld
- t.co
- .telegram.tld
- .tiny.tld
- .tinyurl.tld
- .tumblr.tld
- .twimg.tld
- .twitch.tld
- .twitter.tld
- .uppersafe.tld
- .vk.tld
- .weibo.tld
- .whatsapp.tld
- .wix.tld
- .yahoo.tld
- .ymail.tld
- .youku.tld
- youtu.be
- .youtube.tld
- .ytimg.tld
- .yy.tld
- .wetransfer.tld
9 changes: 9 additions & 0 deletions daemon/__init__.py
@@ -0,0 +1,9 @@
#!/usr/bin/env python3
##
# Nicolas THIBAUT
# nicolas.thibaut@uppersafe.com
##
# -*- coding: utf-8 -*-

from .sensor import sensor
from .syncfw import syncfw
62 changes: 62 additions & 0 deletions daemon/sensor.py
@@ -0,0 +1,62 @@
#!/usr/bin/env python3
##
# Nicolas THIBAUT
# nicolas.thibaut@uppersafe.com
##
# -*- coding: utf-8 -*-

from utils import configuration, database, logger
import sys, re, time, os

conf = configuration()
db = database(conf.get("db"))
log = logger(__name__, conf.get("debug"))

class sensor:
def __init__(self):
self.ino = 0
self.idx = 0
self.start()

def parse(self, buffer):
for source, destination, port in re.findall(".* SRC=([0-9a-f:.]+) DST=([0-9a-f:.]+) .* DPT=([0-9]+) .*", buffer, re.IGNORECASE):
log.info(str("Blocking '{}' -> '{}:{}'").format(source, destination, port))
db.session.add(db.models.events(source=source, destination=destination, port=port))
try:
db.session.commit()
except Exception as error:
db.session.rollback()
log.error(error)
return 0

def watch(self, file):
self.idx = 0
if not self.ino:
self.idx = os.stat(file).st_size
self.ino = os.stat(file).st_ino
log.info(str("[!] Subscribing to file '{}'").format(file))
with open(file) as fd:
fd.seek(self.idx)
try:
while self.ino == os.stat(file).st_ino:
for element in [x for x in fd.read().split("\n") if x.strip()]:
self.parse(element)
time.sleep(1)
except:
pass
log.warning(str("[!] Unsubscribing from file '{}' (probably because of log rotation)").format(file))
return 0

def start(self):
while self.idx >= 0:
try:
conf.reload()
except Exception as error:
log.critical(error)
return 1
if os.path.exists(conf.get("monitor")):
self.watch(conf.get("monitor"))
else:
log.error(str("[!] Could not find '{}' to monitor network events").format(conf.get("monitor")))
time.sleep(1)
return 0

0 comments on commit d64fc9c

Please sign in to comment.