Added documentation for Read the Docs

CMakeLists.txt

@@ -7,7 +7,7 @@ project(yaramod CXX)
 
 # Configurable options.
 option(YARAMOD_TESTS    "Build tests for yaramod" OFF)
-option(YARAMOD_DOC      "Build doxygen documentation for yaramod" OFF)
+option(YARAMOD_DOCS     "Build doxygen documentation for yaramod" OFF)
 option(YARAMOD_PYTHON   "Build Python extension" OFF)
 option(YARAMOD_EXAMPLES "Build examples" OFF)
 
@@ -43,6 +43,6 @@ add_subdirectory(src)
 if(YARAMOD_TESTS)
 	add_subdirectory(tests)
 endif()
-if(YARAMOD_DOC)
-	add_subdirectory(doc)
+if(YARAMOD_DOCS)
+	add_subdirectory(docs)
 endif()

docs/CMakeLists.txt

@@ -0,0 +1 @@
+add_subdirectory(doxygen)

docs/rtd/.gitignore

@@ -0,0 +1 @@
+_build/

docs/rtd/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/rtd/conf.py

@@ -0,0 +1,57 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'yaramod'
+copyright = '2020, Avast'
+author = 'Avast'
+
+# The full version, including alpha/beta/rc tags
+release = 'v3.3.4'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx_rtd_theme',
+    'sphinx_tabs.tabs'
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']

docs/rtd/examples.rst

@@ -0,0 +1,17 @@
+========
+Examples
+========
+
+There are 2 examples available in the repository as of now. They are both implemented in C++ and Python so that you can check out the language which you prefer.
+
+Dump rule AST
+=============
+
+Dump rule AST example shows you how you can implement traversal of the whole AST of the YARA rule condition. It prints out each node it visits and dumps some information to the output.
+
+Boolean simplifier
+==================
+
+Boolean simplifier shows you how you can implement visitor which also changes the condition based on some kind of analysis. In this example, it tries to evaluate the logical operators
+``and``, ``or`` and ``not`` and simplify the condition. It is built on the fact that ``<anything> and false = false`` and ``<anything> or true = true``. Then it just uses typical
+truth tables for ``and``, ``or`` and ``not``.

docs/rtd/formatting_rulesets.rst

@@ -0,0 +1,45 @@
+Formatting rulesets
+===================
+
+Our recent addition to the things yaramod should do is automatic formatting of YARA rules into form which is mostly used for our YARA rules and in which we expect YARA rules when they come from other sources.
+Having formatted YARA rules has same importance as having properly formatted code. It is readable even if you are not the original author of the ruleset and also formatting gives additional semantics without explicitly stating it if you are used to it.
+Formatting that is designed well can also help you catch problems in your ruleset even before you actually submit it and start using it. Therefore we came up with this formatting best to suite our needs which incorporates:
+
+* Use of TABs
+* ``{`` and ``}`` on their own lines
+* ``meta``, ``strings`` and ``condition`` indented once
+* Their contents idented twice
+* New lines when using ``and``, ``or``, ``(`` and ``)`` in condition
+* Comments that are on the consecutive lines are properly aligned
+* Single space in every other place wher you can put multiple whitespaces
+
+Here is an example of the rule before and after formatting:
+
+.. code::
+
+  rule abc {
+  strings:
+    $s01  = "Hello"
+    $s02  = "World"
+  condition:
+    $s01 and $s02
+  }
+
+.. code::
+
+  // This is my rule
+  rule abc
+  {
+          strings:
+                  $s01 = "Hello" // String 1
+                  $s02 = "World" // String 2
+          condition:
+                  $s01 and
+                  $s02
+  }
+
+
+In order to use it access ``getTextFormatted()`` (``text_formatted`` in Python) on YARA file. It is important to state that our formatter **does not remove comments**.
+
+This feature is brand new and is still evolving. We would really appreciate your feedback here so we can improve it further. There is a `staging area <https://github.com/avast/yaramod/wiki/Autoformatting>`_ on our wiki where we collect ideas
+on where to go with autoformatting. Just file an issue if you have any remarks.

docs/rtd/getting_started.rst

@@ -0,0 +1,62 @@
+===============
+Getting Started
+===============
+
+Before we start, we would like to introduce some basic concepts we use in yaramod so you can get familiar with them and know how to use them.
+
+Yaramod as a library serves two main purposes:
+
+* Parsing of YARA rulesets into intermediate form which is friendly for inspection, analysis or even transformations
+* Creation of new YARA rulesets through code using declarative description of how your YARA rule should look like
+
+These two together form a strong tooling for you to build more advanced systems, perform different kind of analyses over your rulesets, etc.
+
+We have our own custom parser for YARA, so we don't use the one in `VirusTotal/yara <https://github.com/VirusTotal/yara>`_, but we try to stay consistent as much as
+we can. Generally, you shouldn't run into situation where YARA does parse your rule and yaramod does not (or vice versa) but this should be considered as bug and should be reported
+to us using `issue tracker <https://github.com/avast/yaramod/issues>`_ on GitHub.
+
+It is also important to state that yaramod in no way does any kind of matching because it does not understand your ruleset in a sense what you want to match.
+Yaramod just knows what is inside of your rules and gives you an option to process it. Despite that, yaramod performs semantic checks where possible so you
+for example need to provide correct types of parameters to module function calls or you have to import a module in order to use it etc.
+
+Modules
+=======
+
+YARA is easily extensible with modules which provide you way to call C functions from your YARA ruleset. We realize that modules are important
+and yaramod therefore supports them. You might however run into functions which you do not recognize and can' find anywhere in upstream YARA.
+We at Avast use YARA daily and sometimes there are things we would like to match in our YARA rules something what is not accessible to the outside
+world so we improve existing or develop our own modules. What we can we share with the community and in the future, we would really like to see
+all the functions we have also in upstream YARA but that doesn't entirely depend on us. It's not like we are keeping it away becuase we want to keep
+it secret.
+
+Our extensions of modules are integrated into yaramod but we realize that you might want to use this library even without our symbols and therefore
+there is a way you can request yaramod to not provide you anything additional.
+
+We have also incorporated this amazing `androguard-yara <https://github.com/androguard/androguard>`_ module which is not merged into upstream of YARA
+but we think it is very cool.
+
+VirusTotal symbols
+==================
+
+YARA allows you to define so called *external variables* which are values outside your YARA environment but they are constant across the whole YARA scan. You can
+reference them in your rule conditions to give an additional data to your rule so it can provide you with the matches you want.
+
+These *external variables* are widely used on VirusTotal as you can see `here <https://www.virustotal.com/intelligence/help/malware-hunting/>`_. We realize this
+and we also realize that you might want to process rulesets that you are using in VirusTotal so we also provide these additional symbols but you can opt-out of this
+feature if you want.
+
+Yaramod instance
+================
+
+The entrypoint of all yaramod is class ``Yaramod`` which you should instantiate and keep it alive while you are doing anything with yaramod. Accessing internal representation
+of YARA rules which were returned by ``Yaramod`` is completely unsafe and can lead to crashes of your application if you do it after ``Yaramod`` has been destructed. Creation of ``Yaramod`` object
+is performance heavy so you should keep the amount of instances low (ideally just one). ``Yaramod`` itself is not thread-safe, so in parallel environment we would suggest
+you to create one instance per thread or process.
+
+``Yaramod`` accepts one optional parameter when creating it and that is *import features*. This option specify in what kind of rulesets you are interested in and you can choose from:
+
+* *All current* - This is the default option which provides you with both Avast-specific and VirusTotal-specific symbols.
+* *Everything* - This also includes deprecated functions which should no longer be used.
+* *Basic* - This represents that you are not interested in any additional symbols in your rules.
+* *Avast* - You are interested in basic and Avast-specific symbols.
+* *VirusTotal* - You are interested in basic and VirusTotal-specific symbols.

docs/rtd/index.rst

@@ -0,0 +1,19 @@
+.. yaramod documentation master file, created by
+   sphinx-quickstart on Fri Feb 21 12:31:51 2020.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to yaramod's documentation!
+===================================
+
+Welcome to the documentation of our project Yaramod. Parser and builder for YARA rulesets.
+
+.. toctree::
+   :maxdepth: 2
+
+   installation
+   getting_started
+   parsing_rulesets
+   creating_rulesets
+   formatting_rulesets
+   examples

docs/rtd/installation.rst

@@ -0,0 +1,53 @@
+============
+Installation
+============
+
+Requirements
+============
+
+In order to install ``yaramod`` you will need:
+
+* `CMake <https://cmake.org/>`_ 3.6+
+* C++ compiler with C++17 support
+
+If you want to use Python bindings then you will also need:
+
+* Python 3.6+
+
+Compilation
+===========
+
+Yaramod is primarily written in C++ so you'll have to compile it to the library which you can then link to your own project. We do not provide any prebuild binary releases so if you
+would like to use it from C++ code, you need to compile it yourself. We use `CMake <https://cmake.org/>`_ for our build system so you'll need to install it beforehand. We support
+3 main platforms - Linux, Windows and macOS. The project might work also on other platforms but we do not provide any support for those nor any guarantees that it works.
+
+In order to configure the project and start the compilation run
+
+.. code-block:: bash
+
+  mkdir build
+  cmake -DCMAKE_BUILD_TYPE=<Release|Debug> ..
+  cmake --build .
+
+If you would also like to build examples (located in `src/examples/cpp <https://github.com/avast/yaramod/tree/master/src/examples/cpp>`_) then also pass ``-DYARAMOD_EXAMPLES=ON`` to the
+second command, right after ``-DCMAKE_BUILD_TYPE=...``.
+
+It is also possible to build Python bindings directly from CMake by providing an option ``-DYARAMOD_PYTHON=ON``.
+
+Python Bindings
+===============
+
+Yaramod also comes with Python 3 bindings in case you don't want to deal with linking C++ libraries or you just simply prefer Python over C++. We deploy yaramod to PyPI so you can
+install it directly from there using ``pip``.
+
+.. code-block:: bash
+
+    pip install yaramod
+
+In case you don't want to use the latest release but would much rather use ``master`` branch you can run
+
+.. code-block:: bash
+
+    pip install .
+
+from the root directory of yaramod directly.

docs/rtd/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd