Docs for 'wire' data description and backdoor architecture

MANIFEST

@@ -3,7 +3,8 @@ setup.py
 covertutils/__init__.py
 covertutils/exceptions.py
 covertutils/helpers.py
-covertutils/payloads.py
+covertutils/bridges/__init__.py
+covertutils/bridges/simplebridge.py
 covertutils/crypto/__init__.py
 covertutils/crypto/algorithms/__init__.py
 covertutils/crypto/algorithms/cyclingalgorithm.py
@@ -28,13 +29,40 @@ covertutils/handlers/resettable.py
 covertutils/handlers/responseonly.py
 covertutils/handlers/stageable.py
 covertutils/handlers/impl/__init__.py
+covertutils/handlers/impl/extendableshell.py
 covertutils/handlers/impl/simpleshell.py
+covertutils/handlers/impl/standardshell.py
 covertutils/orchestration/__init__.py
 covertutils/orchestration/orchestrator.py
 covertutils/orchestration/simpleorchestrator.py
 covertutils/orchestration/stegoorchestrator.py
 covertutils/orchestration/streamidentifier.py
-covertutils/bridges/__init__.py
-covertutils/bridges/simplebridge.py
+covertutils/payloads/__init__.py
+covertutils/payloads/generic/__init__.py
+covertutils/payloads/generic/control.py
+covertutils/payloads/generic/echo.py
+covertutils/payloads/generic/example.py
+covertutils/payloads/generic/file.py
+covertutils/payloads/generic/info.py
+covertutils/payloads/generic/pythonapi.py
+covertutils/payloads/generic/shell.py
+covertutils/payloads/generic/shellprocess.py
+covertutils/payloads/linux/__init__.py
+covertutils/payloads/linux/shellcode.py
+covertutils/payloads/windows/__init__.py
+covertutils/payloads/windows/shellcode.py
 covertutils/shells/__init__.py
-covertutils/shells/textshell.py
+covertutils/shells/baseshell.py
+covertutils/shells/impl/__init__.py
+covertutils/shells/impl/extendableshell.py
+covertutils/shells/impl/simpleshell.py
+covertutils/shells/impl/standardshell.py
+covertutils/shells/subshells/__init__.py
+covertutils/shells/subshells/controlsubshell.py
+covertutils/shells/subshells/examplesubshell.py
+covertutils/shells/subshells/filesubshell.py
+covertutils/shells/subshells/pythonapisubshell.py
+covertutils/shells/subshells/shellcodesubshell.py
+covertutils/shells/subshells/simplesubshell.py
+covertutils/shells/subshells/stagesubshell.py
+covertutils/stages/http_stage.py

docs/architecture.rst

@@ -0,0 +1,144 @@
+Ingredients for Cooking a `Backdoor`
+====================================
+
+Every program that uses the `covertutils` way of implementing communication has the following ingredients. No matter if it is the backdoors **Agent** (the one running on the compromised host), or the **Handler** (the one that is used by the `pentester`/`attacker`/`hoodie wearing guy`).
+
+All backdoors designed with `coverutils` are made with the below parts:
+
+
+`500gr` - ``Orchestrator``
+--------------------------
+
+Docs @ :class:`covertutils.orchestration.orchestrator.Orchestrator`
+
+This `class` manipulates data from and to the `Communication Channel` (see below).
+When several data transformations are in place, the ``Orchestrator`` objects are the ones that pass them from their `minified` / `encrypted` / `chunked` form to their original, lossless form.
+
+This is done without any interaction from the developer in `packet level` as in `covertutils` there are **no packets**.
+All networking is abstracted, and what is finally traveling in network packets is no concern for `covertutils` classes.
+
+Message Packing
+***************
+
+Packing and unpacking is done by the :meth:`covertutils.orchestration.orchestrator.Orchestrator.readyMessage` `(packing)` and :meth:`covertutils.orchestration.orchestrator.Orchestrator.depositChunk` `(unpacking)` methods.
+
+The ``Orchestrator`` instances are designed to create pairs of 2, in a way that data the first ``Orchestrator`` instance `packs` to `sendable forms` are able to be unpacked and read **only** by the second ``Orchestrator`` instance and `vice-versa`. This is done with the use of a `passphrase` to initialize all `encryption keys` and `stream OTPs`
+
+Streams
+*******
+
+Every ``Orchestrator`` object supports a list of `streams` defined as a `python list` at instance initialization. That means that Messages are packed against a `stream` (falling back to a `default` if not specified) every time the :meth:`readyMessage` is used.
+It is possible to add and remove `streams` at runtime with :meth:`covertutils.orchestration.orchestrator.Orchestrator.addStream` and :meth:`covertutils.orchestration.orchestrator.Orchestrator.deleteStream` methods. This is highly utilized by staging, as stages always operate in different `streams` (see :ref:`stages_page`).
+
+
+Identities
+***********
+
+To ensure compatibility between two ``Orchestrator`` objects an `Identity string` is provided for each ``Orchestrator`` object which is generated by computing hashes of its initial settings (including `passphrase`). Identity strings and compatibility checks for them are provided by the :meth:`covertutils.orchestration.orchestrator.Orchestrator.checkIdentity` and :meth:`covertutils.orchestration.orchestrator.Orchestrator.getIdentity` methods.
+
+
+
+`750gr` - ``Handler``
+---------------------
+
+Docs @ :mod:`covertutils.handlers` `(enlightening)`
+
+
+.. warning:: The ``*Handler`` classes are used to create **both** `Agents` and `Handlers`. Their name derives from their ability to `handle` Messages. Messages are received both ways! So a **Reverse TCP Agent** could consist of a :class:`covertutils.handlers.impl.simpleshell.SimpleShellHandler` object to let it communicate with the other side and `shell_exec` all incoming `Messages`.
+
+.. note:: Think of ``*Handler`` objects like `sockets` with **callbacks**.
+
+The ``*Handler`` classes are used along with ``Orchestrators`` to provide an interface to the developer for 2 things:
+
+ - How the received `Messages` are used (displayed? - executed? - stored in files?)
+ - How the `Agent` or `Handler` will respond on Messages, and the `general Behavior` of it.
+
+
+
+`Messages`
+**********
+
+The API described in Docs @ :mod:`covertutils.handlers` totally abstract the whole raw-data to `Message`, `Stream` transformations, setting callbacks that are meaningful to the receiver.
+
+The callbacks run when:
+
+ - a `Message` has arrived, informing about the `Message`'s content and the `Stream` it received it - :meth:`covertutils.handlers.basehandler.BaseHandler.onMessage`
+ - a `Chunk` has arrived, informing about the `Stream` that it received it, and if it was the last part of a `Message` - :meth:`covertutils.handlers.basehandler.BaseHandler.onChunk`
+ - an unrecognized payload has arrived - :meth:`covertutils.handlers.basehandler.BaseHandler.onNotRecognised`
+
+
+Behaviors
+*********
+
+At time of writing 2 behaviors have been identified in the wild and modeled.
+The `interrogating` one and the `bind'ish` one.
+
+
+Interrogating
++++++++++++++
+
+It is when one of the two sides is periodically querying the other side. This is the behavior of all reverse HTTP/S backdoor `Agents`.
+
+Yet, it has not been hardcoded for use with `Agents` only! A handler could be interrogating too. Think of an ICMP backdoor Handler. If the `Agent` has to transfer a huge response back to the `Handler`, The Handler has to start an `interrogating` process for the `Agent` to respond with payload chunks, until the whole `Message` gets across. This is the only way to resemble a ping-pong behavior.
+
+That behavior is modeled in the :class:`covertutils.handlers.interrogating.InterrogatingHandler` class, and it is used in the :ref:`icmp_bind_example` from the `Handler` exactly as described above.
+
+
+
+`Bind'ish`
+++++++++++
+
+This behavior is found when the `one side` **MUST NOT TALK** under any circumstance, **unless asked** by the `other side`. Most of the time it is just the **complementary** of the `Interrogating` behavior. This is the case for `Reverse HTTP/S backdoors`
+
+
+We find this behavior typically in `HTTP/S reverse Handlers`. As `HTTP/S reverse Handlers` act as (or actually are_ ) `HTTP/S Servers`, usually also serving error pages just to be more persuading for their innocence.
+
+The thing is that a presumable HTTP/S Server **never sends things to an HTTP Client before getting an HTTP request**. So the :class:`covertutils.handlers.responseonly.ResponseOnlyHandler` keeps a list of stuff that has to transmit and sends them over **only if** a `request Message arrives`.
+
+
+
+Both `Interrogating` and `Bind'ish` behaviors can use the :meth:`covertutils.handlers.basehandler.BaseHandler.sendQueue` method to send Messages over.
+
+.. _are: https://github.com/EmpireProject/Empire/blob/master/empire
+
+Ad Hoc Behavior
++++++++++++++++
+
+That is all good but a good `ol' Reverse TCP` needs none of them! For such cases the :meth:`covertutils.handlers.basehandler.BaseHandler.sendAdHoc` method saves the day. Just spits to the `Communication Channel` like there is no tomorrow (or IDS to trick).
+
+Both class methods are available in all behaviors (defined in `base class`) but each one has the `sending method` that fits the behavior that is trying to simulate. The :meth:`covertutils.handlers.basehandler.BaseHandler.preferred_send` always holds the `sending method` best fit for the ``Handler`` insatnce used.
+
+
+
+
+A pinch of `Communication Channel` Creativity
+---------------------------------------------
+
+Networking isn't a standard thing when designing a backdoor. This is why it is left out of the way completely.
+All Communication is wrapped by ``send()`` and ``recv`` functions where the ``send()`` has to get 1 argument (raw data to send) and the ``recv()`` has to be blocking.
+
+.. code :: python
+
+	to_handler = []
+	to_agent = []
+
+	#=========================================
+	def handler_send(raw_data) :
+		to_agent.append(raw_data)
+
+	def handler_recv() :
+		while not to_handler: pass	# Has to block when no payloads arrive
+		return to_handler.pop(0)
+
+	#=========================================
+	def agent_send(raw_data) :
+		to_handler.append(raw_data)
+
+	def agent_recv() :
+		while not to_agent: pass	# If you are brave enough use queues and threads
+		return to_agent.pop(0)
+
+This is perfectly working example of wrapping functions. It is actually really useful for testing stuff.
+
+
+So those functions can use ``requests`` to post to `pastebin` or do whatever. The `covertutils` package doesn't care about **how** you get your bytes to the other side. It just guarantees that the bytes will be fully unrecognizable (see: :ref:`ids_evasion`) to anyone else than the other side.

docs/components.rst

@@ -1,5 +1,5 @@
-Components
-===========
+Internal Components
+===================
 
 Here you can find code snippets from *covertutils* basic internal components.
 They are documented under the :mod:`covertutils` pages, but here they will parade in all their glory.

docs/ids_evasion.rst

@@ -0,0 +1,71 @@
+
+.. _ids_evasion:
+
+Total IDS/IPS evading payloads
+==============================
+
+Whatever travels from and to ``Handler`` classes is generated using ``Orchestrator`` instances. That means that, not only the communication is encrypted, but there are no `moving parts` on the what is transmitted too (`I will elaborate`).
+
+
+`No-Duplicate` Principle
+------------------------
+
+For making the protocol payloads hard to identify I implemented what (I self named) the `No-Duplicate` principle.
+
+This assures that every two consecutive payloads (or more in fact) have the same bytes in same position with a probability of `1/512` (i.e. completely randomly).
+
+In plain english, if I issue an ``ls -l`` `(5 bytes without new line)` command and the `3rd byte` of the payload generated happens to be ``\x67`` (due to encryption it probably won't be ``\x20`` `-space character-` anyway), this `Principle` says that if I issue again ``ls -l``, the 3rd byte has a 1/512 probability of being ``\x67`` again.
+
+
+Implementation Pitfalls
+-----------------------
+
+**DUH!** ::
+
+	The "No-Duplicate Principle" generally applies to all Stream Ciphers.
+	So, as I use a (homebrew) Stream Cipher,
+	I got that principle covered too. Right?
+
+
+**Right. But...** this is tricky to implement for the **whole protocol**. That is **from first payload generated** and for **every payload**.
+
+And the tricky part is that if the payload is **not tagged in any way** it is difficult for the listener to determine whether the received data is addressed to him and **not try to decrypt all kinds of received data**.
+
+Making the listener **identify** whether the decrypted data is gibberish (and rollback the key if it is) will need to provide a concise definition of **what gibberish is**. And doing even that (`dangerous high entropy approach`) will disable the sender from **sending gibberish intentionally**. Not bright idea at all. `Crypted shellcodes` **look like gibberish** but I can see reasons for sending such things...
+
+*Decrypting data that is not addressed to the listener is a big problem if a Stream Cipher is used. It makes one of the keys to cycle `without notifying the other side`, `ultimately scraping` the rest of the connection.
+
+
+And that is as the data doesn't get transmitted through a known connection. TLS application data seem random too, but it travels through a TCP connection that **knows how to handle them because of the handshake**.
+
+*In a backdoor a handshake will generate `signatures`, as any hardcoded `byte-position pair`.
+
+
+And using a legit protocol like TLS would hardcode the `Network Agnostic` design all together. Yet TLS can be used, if wrapped with `covertutils` functions, but making the only option is far from useful.
+
+So you see. It is indeed tricky...
+
+
+
+OTP me to the `End of Love`
+---------------------------
+
+So, as using a single byte signature is a big **NO-NO** what stands out?
+
+What if we encrypt a string with a Rolling Key and append it to the message?
+
+As the key is rolling the "`No-Duplicate Principle`" applies to that encrypted string. But no the listener **know what to search for**. Decrypting a certain portion of the payload will have to result to the original string. If it does not then the received data is of no interest for the listener (the sender didn't sent this one), and the key can be rolled back.
+
+This is a kind of one time password (`OTP`) for each payload originally sent from the sender.
+
+This mechanism is furtherly described in the Securosophy_ post, under the `How Streams are implemented` heading.
+
+.. _Securosophy : https://securosophy.com/2017/04/22/reinventing-the-wheel-for-the-last-time-the-covertutils-package/
+
+Still, there is a possibility for a random data packet that have been received to have the same decrypted value with that string and this will mean that the channel will hang. That possibility is more that finding a `Shiny Pokemon in Silver`__, so it is handled (`how` is also explained in the `blog post` above).
+
+
+.. _celebi : https://i.ytimg.com/vi/O7ZsJV71ji0/maxresdefault.jpg
+
+
+__  celebi_

docs/index.rst

@@ -12,7 +12,7 @@ Welcome to `CovertUtils`'s documentation!
 
 This Project is free and open-source, available @ Github_
 
-.. _Github : https://github.com/operatorequals/covertutils.
+.. _Github : https://github.com/operatorequals/covertutils
 
 
 A Blog post about it, explaining *motivation* and *implementation internals* is located in my personal blog: Securosophy_
@@ -26,6 +26,7 @@ Not a Backdoor!
 Well, `almost` not a backdoor. This project is a `Python2 package` containing enough modules for implementing custom backdoors.
 Everything, from file transferring to customized shells are included.
 
+
 It is not a backdoor ready to be planted (well, most of the :ref:`programming_examples` are). If you are looking for backdoors, RATs and such stuff in `Python` there are some awesome projects already:
 
  - Weevely_
@@ -74,24 +75,39 @@ And all that with the abstraction of **Object Oriented Programming**, as this pa
 
 .. toctree::
  :maxdepth: 1
- :caption: Contents:
+ :caption: Topics:
 
  installation
  package_structure
  components
+ architecture
  prog_examples
+ ids_evasion
  shells
  native_execs
  stages
  stage_api
 
+
+All modules `[citation needed]` are documented automatically from comments with `Sphinx apidoc`. The output is below...
+
 .. toctree::
- :maxdepth: -1
- :caption: Basic Blocks:
+ :maxdepth: 3
+ :caption: The whole "apidoc" pages ahead:
+
+ modules
+ covertutils
+
+
+
+
+As the *covertutils API* Toc-Tree is **huge** (due to the code organizing, see: :ref:`package_structure`), it is really handy to use the **search page** of **Sphinx** if you are looking for a specific `class` or `method`.
+
+
+
 
- covertutils.handlers
- covertutils.orchestration
 
+.. note:: For `flawless backdoor creation` don't forget to fire up some Primus CDs or old blues standards while coding. Maybe light a cigar too.
 
 
-As the *covertutils API* Toc-Tree is **huge** (due to the code organizing, see: :ref:`package_structure`), it is really handy to use the **search page** of **Sphinx**.
+.. note:: Creating stealthy backdoors requires intelligence, and intelligence is **a terrible thing to waste**.

docs/prog_examples.rst

@@ -118,6 +118,7 @@ Traffic Sample
 
 
 
+.. _icmp_bind_example:
 
 Advanced ICMP Bind Shell
 ---------------------------

docs/shells.rst

@@ -73,7 +73,7 @@ This way SubShells with only one or two commands can be used without directly ac
 	(127.0.0.1:58504)> !control reset
 
 
-*The ``"!"`` can be used from SubShells too, making file transfers handy.
+The ``"!"`` can be used from SubShells too, making file transfers handy:
 
 .. code:: bash
 

docs/stages.rst

@@ -212,8 +212,8 @@ The `covertutils` package has a `file` stage and subshell, to provide file trans
 	=|file]> ~ File uploaded succesfully!
 
 
-*Providing file transfer `in-band` is a double-edged knife.
+.. warning:: Providing file transfer `in-band` is a double-edged knife.
 If the `Communication Channel` is a TCP connection then files will flow around nicely (taking also advantage of the embedded compression, see: :ref:`compressor_component` ).
 But if the `Communication Channel` is a `covert TCP backdoor` or such `super-low-bandwidth` channel, a 1MB file will `take forever to download`, taking over the whole channel. An out-of-band approach should be considered in this case.
 
-*Transfer of files can trigger the :class:`StreamIdentifier`'s `Birthday Problem` (TODO: document it) destroying 1 or more `streams` (the `control stream` should still work to ``!control reset`` the connection). For heavy use of file transferring, a bigger ``tag_length`` should be used on the :class:`Orchestrator` passed to the :class:`Handler` object.
+.. warning:: Transfer of files can trigger the :class:`StreamIdentifier`'s `Birthday Problem` (TODO: document it) destroying 1 or more `streams` (the `control stream` should still work to ``!control reset`` the connection). For heavy use of file transferring, a bigger ``tag_length`` should be used on the :class:`Orchestrator` passed to the :class:`Handler` object.

makefile

@@ -12,7 +12,7 @@ test :
 
 doc :
 	# cd docs/;
-	sphinx-apidoc ./covertutils/ -P -f -o docs/
+	sphinx-apidoc ./covertutils/ -P -f  -o docs/
 	cd docs/;make html
 
 

setup.py

@@ -1,5 +1,5 @@
-from distutils.core import setup
-# from setuputils import setup
+# from distutils.core import setup
+from setuptools import setup
 
 import covertutils