Refactor candidate 2 (#75)

# 1.4.0:
* Refactor FakeNet-NG to unify Windows and Linux packet handling
* Remove Proxy Listener UDP stream abstraction to prevent issue where
  subsequent clients do not receive response packets because the proxy listener
  continues to send them to the old (expired) ephemeral port for the previous
  client
* Stop flag command-line support for rudimentary IPC-based start/stop
  automation
* Integration test script for MultiHost and SingleHost mode
* Fixed Errno 98 (`TIME_WAIT`) issue with `RawTcpListener`
* WinDivert `GetLastError` exception work-around for [WinDivert issue
  #32](ffalcinelli/pydivert#32)

docs/changes.md

@@ -0,0 +1,14 @@
+# Change Log
+
+## 1.4.0:
+* Refactor FakeNet-NG to unify Windows and Linux packet handling
+* Remove Proxy Listener UDP stream abstraction to prevent issue where
+  subsequent clients do not receive response packets because the proxy listener
+  continues to send them to the old (expired) ephemeral port for the previous
+  client
+* Stop flag command-line support for rudimentary IPC-based start/stop
+  automation
+* Integration test script for MultiHost and SingleHost mode
+* Fixed Errno 98 (`TIME_WAIT`) issue with `RawTcpListener`
+* WinDivert `GetLastError` exception work-around for [WinDivert issue
+  #32](https://github.com/ffalcinelli/pydivert/issues/32)

docs/contributors.md

@@ -0,0 +1,35 @@
+# Contributors
+
+This document credits those who conceptualized and/or implemented features for
+FakeNet-NG.
+
+## Legacy
+
+FakeNet-NG is based on the original
+[FakeNet](https://practicalmalwareanalysis.com/fakenet/) tool developed by
+Andrew Honig and Michael Sikorski, which is still the tool of choice for
+malware analysis on Windows XP.
+
+## Windows
+
+Peter Kacherginsky [implemented
+FakeNet-NG](https://www.fireeye.com/blog/threat-research/2016/08/fakenet-ng_next_gen.html)
+targeting modern versions of Windows.
+
+## Linux and Core
+
+Michael Bailey [implemented FakeNet-NG on
+Linux](https://www.fireeye.com/blog/threat-research/2017/07/linux-support-for-fakenet-ng.html),
+and later refactored FakeNet-NG to use this as the unified packet processing
+logic for both Windows and Linux.
+
+## Content-Based Protocol Detection
+
+The original FakeNet-NG was able to automatically handle SSL; meanwhile, Joshua
+Homan developed the original concept of using a protocol "taste" callback to
+sample traffic and direct clients to the appropriate server ports. Matthew
+Haigh, Michael Bailey, and Peter Kacherginsky conceptualized the Proxy Listener
+and Hidden Listener mechanisms for introducing both of these content-based
+protocol detection features to FakeNet-NG. Matthew Haigh then [implemented
+Content-Based Protocol
+Detection](https://www.fireeye.com/blog/threat-research/2017/10/fakenet-content-based-protocol-detection.html).

docs/internals.md

@@ -1,15 +1,9 @@
 # FakeNet-NG Internals
 
-## FakeNet-NG Linux Diverter Internals
+This documentation was originally written for the Linux implementation, and
+where specifics are called for, it currently references Linux.
 
-The FakeNet-NG Diverter implementation for Linux uses
-[netfilter](http://netfilter.org/) to examine, log, and redirect packets.
-Netfilter is available at the kernel level through loadable kernel modules
-(LKMs) and in userspace through the
-[libnetfilter_queue](http://netfilter.org/projects/libnetfilter_queue/index.html)
-library. Since FakeNet-NG is Python-based, it uses libnetfilter_queue via the
-[python-netfilterqueue](https://github.com/kti/python-netfilterqueue/) wrapper
-which is written in Cython.
+## FakeNet-NG Diverter Internals
 
 For purposes of this documentation, some rigorously-defined terms will be
 repurposed or replaced:
@@ -21,6 +15,20 @@ repurposed or replaced:
   we will use the more general term _conversation_ to represent the concept of
   a pair of endpoints (again, using that term loosely) that are communicating.
 
+FakeNet-NG can also operate in two modes on Linux:
+* `SingleHost` - simulate the Internet for the local machine
+* `MultiHost` - simulate the Internet by acting as the gateway for another
+  machine
+
+Each implementation of FakeNet-NG ultimately relies on a driver or kernel
+module that supports network hooking and a library that makes this accessible
+from user space. The Windows Diverter uses
+[PyDivert](https://github.com/ffalcinelli/pydivert) to control the
+[WinDivert](https://reqrypt.org/windivert.html) driver. The Linux Diverter uses
+[python-netfilterqueue](https://github.com/kti/python-netfilterqueue) to access
+[libnetfilter_queue](https://netfilter.org/projects/libnetfilter_queue/) and in
+turn [NetFilter](https://netfilter.org/).
+
 ### Traffic Flow Condition Evaluation
 
 The simplest case for the Linux implementation of the FakeNet-NG Diverter is
@@ -30,10 +38,9 @@ we use `iptables` to implement a `REDIRECT` rule in the `PREROUTING` chain.
 In this use case, FakeNet-NG implements only dynamic port forwarding (DPF)
 using python-netfilterqueue.
 
-The most complicated case is `SingleHost` mode (experimental), in which both
-DPF and NAT must be controlled by FakeNet-NG to permit process blacklisting
-and other configuration settings. In this case, FakeNet-NG uses
-python-netfilterqueue to evaluate four conditions:
+The more complicated case is `SingleHost` mode, in which both DPF and NAT must
+be controlled by FakeNet-NG to permit process blacklisting and other
+configuration settings. In this case, FakeNet-NG evaluates four conditions:
 1. When a packet is produced, is it destined for a foreign IP address? (if so,
   fix up its destination address to be a local address)
 2. When a packet is about to be consumed, is it destined for an unbound port?
@@ -45,8 +52,8 @@ python-netfilterqueue to evaluate four conditions:
   that has been NATted? (if so, fix up its source IP)
 
 Given two processes `P1` and `P2`, here is a diagram of communication and
-condition evaluation using the `INPUT` and `OUTPUT` chains provided by
-Netfilter:
+condition evaluation specific to Linux, using the `INPUT` and `OUTPUT` chains
+provided by Netfilter:
 
 ```
          (1)                                                      (2)
@@ -288,7 +295,6 @@ Or, in Python:
 				(not sport_bound and not dport_bound))
 ```
 
-
 ### Future
 
 #### NetworkMode Auto for Linux
@@ -312,11 +318,12 @@ troubleshooting problems transferring the 24KB file `FakeNet.gif` over FTP.
 
 This is fine for `MultiHost` mode because external interfaces (e.g. `eth0`)
 frequently have a maximum transmittal unit (MTU) of 1500. However, for loopback
-communications where the MTU is 65536, this causes errors. It is possible to
-fix these errors by changing the buffer size to 65616, however this may be
-overridden by future installations of python-netfilterqueue either via the
-package management system specific to the Linux distribution, Pip, etc.
+communications where the MTU may be something like 65536, this causes errors.
+It is possible to fix these errors by changing the buffer size to 65616
+(accounting for 80 bytes of overhead), however this could be overridden by
+future installations of python-netfilterqueue either via the package management
+system specific to the Linux distribution, Pip, etc.
 
 A work-around for this issue is to send all NAT packets through an externally
-facing IP address instead of 127.0.0.1 to avoid exposing ourselves to
-`BufferSize < MTU` conditions such as in the transfer of large files.
+facing IP address instead of 127.0.0.1 to avoid exposing traffic to `BufferSize
+< MTU` conditions such as in the transfer of large files.

fakenet/configs/default.ini

@@ -26,9 +26,26 @@ DivertTraffic:         Yes
 # NetworkMode:           MultiHost
 NetworkMode:           Auto
 
-# DebugLevel (Linux only as of this writing): specify fine-grained debug print
-# flags to enable. Enabling all logging when verbose mode is selected results
-# in an unacceptable overhead cost, hence this setting.
+# DebugLevel: specify fine-grained debug print flags to enable. Enabling all
+# logging when verbose mode is selected results in overwhelming output, hence
+# this setting. Valid values (comma-separated) are:
+# 
+#    GENPKT     Generic packet information
+#    GENPKTV    Packet analysis, displays IP, TCP, UDP fields, very wide output
+#    CB         Diverter packet handler callback start/finish logging
+#    NONLOC     Nonlocal packet verbose logging
+#    DPF        Dynamic port forwarding decisions
+#    DPFV       Dynamic port forwarding table activity
+#    IPNAT      NAT decisions
+#    MANGLE     Packet mangling (modification) activity
+#    PCAP       PCAP writes of original and mangled packets
+#    IGN        Cases where packets are forwarded as is
+#    FTP        FTP-specific logic
+#    IGN-FTP    Cases where packets are forwarded as is due to FTP Active Mode
+#    MISC       Miscellaneous
+#    NFQUEUE    NetfilterQueue activity (Linux only)
+#    PROCFS     Procfs read/write activity (Linux only)
+#    IPTABLES   iptables firewall rule activity (Linux only)
 DebugLevel:            Off
 
 # MultiHost mode only: Specify what interfaces the Linux Diverter should create

fakenet/diverters/debuglevels.py

@@ -0,0 +1,42 @@
+# Debug print levels for fine-grained debug trace output control
+DNFQUEUE = (1 << 0)     # netfilterqueue
+DGENPKT = (1 << 1)      # Generic packet handling
+DGENPKTV = (1 << 2)     # Generic packet handling with TCP analysis
+DCB = (1 << 3)          # Packet handlign callbacks
+DPROCFS = (1 << 4)      # procfs
+DIPTBLS = (1 << 5)      # iptables
+DNONLOC = (1 << 6)      # Nonlocal-destined datagrams
+DDPF = (1 << 7)         # DPF (Dynamic Port Forwarding)
+DDPFV = (1 << 8)        # DPF (Dynamic Port Forwarding) Verbose
+DIPNAT = (1 << 9)       # IP redirection for nonlocal-destined datagrams
+DMANGLE = (1 << 10)     # Packet mangling
+DPCAP = (1 << 11)       # Pcap write logic
+DIGN = (1 << 12)        # Packet redirect ignore conditions
+DFTP = (1 << 13)        # FTP checks
+DMISC = (1 << 27)       # Miscellaneous
+
+DCOMP = 0x0fffffff      # Component mask
+DFLAG = 0xf0000000      # Flag mask
+DEVERY = 0x0fffffff     # Log everything, low verbosity
+DEVERY2 = 0x8fffffff    # Log everything, complete verbosity
+
+DLABELS = {
+    DNFQUEUE: 'NFQUEUE',
+    DGENPKT: 'GENPKT',
+    DGENPKTV: 'GENPKTV',
+    DCB: 'CB',
+    DPROCFS: 'PROCFS',
+    DIPTBLS: 'IPTABLES',
+    DNONLOC: 'NONLOC',
+    DDPF: 'DPF',
+    DDPFV: 'DPFV',
+    DIPNAT: 'IPNAT',
+    DMANGLE: 'MANGLE',
+    DPCAP: 'PCAP',
+    DIGN: 'IGN',
+    DFTP: 'FTP',
+    DIGN | DFTP: 'IGN-FTP',
+    DMISC: 'MISC',
+}
+
+DLABELS_INV = {v.upper(): k for k, v in DLABELS.iteritems()}