WIP

docs/conf.py

@@ -23,6 +23,9 @@
 from mock import Mock as MagicMock
 from sphinx import apidoc
 
+import recommonmark
+from recommonmark.parser import CommonMarkParser
+from recommonmark.transform import AutoStructify
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -121,8 +124,14 @@ def __add__(self, other):
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
-# The suffix of source filenames.
-source_suffix = '.rst'
+# Enable markdown parser.
+source_parsers = {
+    '.md': CommonMarkParser,
+}
+
+# The suffixes of source filenames.
+
+source_suffix = ['.rst', '.md']
 
 # The encoding of source files.
 # source_encoding = 'utf-8-sig'
@@ -348,8 +357,16 @@ def RunSphinxAPIDoc(_):
   # sys.path.append(os.path.join(os.path.dirname(__file__)))
   current_directory = os.path.abspath(os.path.dirname(__file__))
   module = os.path.join(current_directory,"..","plaso")
-  apidoc.main(['-o', current_directory, module, '--force'])
+  api_directory = os.path.join(current_directory, 'sources', 'api')
+  apidoc.main(['-o', api_directory, module, '--force'])
 
 def setup(app):
   """Override Sphinx setup to trigger sphinx-apidoc."""
-  app.connect('builder-inited', RunSphinxAPIDoc)
+  # app.connect('builder-inited', RunSphinxAPIDoc)
+  app.add_config_value('recommonmark_config', {
+#    'enable_auto_toc_tree': True,
+#    'auto_toc_tree_section': 'Contents',
+    'enable_auto_doc_ref': True,
+    'url_resolver': lambda x: x,
+  }, True)
+  app.add_transform(AutoStructify)

docs/index.rst

@@ -1,18 +1,31 @@
-Welcome to the Plaso API documentation!
+Welcome to the Plaso documentation!
 =======================================
 Plaso (Plaso Langar Að Safna Öllu) is a computer forensic tool for timeline generation and analysis.
 
-The project's code is available from https://github.com/log2timeline/plaso, and user documentation is available at https://github.com/log2timeline/plaso/wiki/ and http://plaso.kiddaland.com.
+
+Plaso's documentation is split into several parts:
+
+
+-  `User documentation <sources/user/Home.html>`__
+-  `Developer documentation <sources/developer/Home.html>`__
+-  `API documentation <sources/api/plaso.html>`__
+
+
+The project's code is available from https://github.com/log2timeline/plaso
 
 Plaso is licensed under the Apache license version 2.
 
-Project Contents:
+Contents:
 
 .. toctree::
    :maxdepth: 3
 
-   modules
+   sources/user/index
+
+.. toctree::
+   :maxdepth: 3
 
+   sources/developer/index
 
 
 Indices and tables

docs/requirements.txt

@@ -3,6 +3,8 @@ pyparsing
 # We have several parsers that create construct objects, and if we install
 # it correctly, the documentation looks better.
 construct
-# We need to pin the version of Sphinx for readthedocs, see
-# https://github.com/rtfd/readthedocs.org/issues/3769
-sphinx==1.7.0
+sphinx==1.7.0
+# Workaround for https://github.com/rtfd/recommonmark/issues/104.
+docutils==0.14
+# Required for markdown support.
+recommonmark

docs/sources/developer/Adding-a-new-dependency.md

@@ -0,0 +1,63 @@
+There are several steps involved for getting a new dependency added to plaso.
+
+## Before you begin checklist
+
+If the answer on any of the questions below is no the dependency is not suitable for plaso. In that case see: [Alternatives](Adding-a-new-dependency#alternatives)
+
+* Has the dependency a [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0) compatible license?
+
+Code licensed under a GPL, AGPL or BSD 4-clause cannot be integrated in a binary build, hence these licenses are deemed incompatible. Note that this could also apply to other source code licenses.
+
+* Does the dependency support Linux, Mac OS X and Windows?
+
+## Automated testing
+
+The automated testing relies on having the dependencies available. We deliberately limit the usage of pip (or equivalent) in the automated testing scripts, mainly because pip distributed builds change continuously. We want to have a more stable set of dependencies on dependencies and limit the amount of troubleshooting due breakage caused by a dependency.
+
+### Getting a dependency in l2tdevtools
+
+Plaso uses [l2tdevtools](https://github.com/log2timeline/l2tdevtools) to limit the amount of manual effort required in maintaining dependencies.
+
+If the build process of the dependency is similar enough what is currently supported then adding a new dependency should be relative straight forward by adding it to [data/projects.ini](https://github.com/log2timeline/l2tdevtools/blob/master/data/projects.ini). If not reach out to discuss how we can get support into l2tdevtools for the specific dependency.
+
+Make sure the dependency is added to the [plaso dependencies preset](https://github.com/log2timeline/l2tdevtools/blob/master/data/presets.ini) for [update.py](https://github.com/log2timeline/l2tdevtools/blob/master/tools/update.py).
+
+### Getting a dependency in GIFT COPR
+
+Once the dependency has been added to l2tdevtools it can be added to the [GIFT COPR](https://copr.fedorainfracloud.org/groups/g/gift/coprs/). Ask one of the [log2timeline maintainers](https://github.com/orgs/log2timeline/teams/log2timeline-maintainers/members) to upload your package.
+
+**Notes for maintainers:** https://github.com/log2timeline/l2tdocs/blob/master/process/GIFT%20COPR.md
+
+### Getting a dependency in GIFT PPA
+
+Once the dependency has been added to l2tdevtools it can be added to the [GIFT PPA](https://launchpad.net/~gift). Ask one of the [log2timeline maintainers](https://github.com/orgs/log2timeline/teams/log2timeline-maintainers/members) to upload your package.
+
+**Notes for maintainers:** https://github.com/log2timeline/l2tdocs/blob/master/process/GIFT%20PPA.md
+
+### Getting a dependency in l2tbinaries
+
+Once the dependency has been added to l2tdevtools it can be added to [l2tbinaries](https://github.com/log2timeline/l2tbinaries). Ask one of the [log2timeline maintainers](https://github.com/orgs/log2timeline/teams/log2timeline-maintainers/members) to upload your package.
+
+**Notes for maintainers:** https://github.com/log2timeline/l2tdocs/blob/master/process/l2tbinaries.md
+
+## Source changes
+
+Dependencies are defined in multiple different configuration files and scripts in the plaso source tree. To keep them consistent plaso uses l2tdevtools update-dependencies.py to generate these configuration files and scripts based on [dependencies.ini](https://github.com/log2timeline/plaso/blob/master/dependencies.ini).
+
+```
+PYTHONPATH=../l2tdevtools ../l2tdevtools/tools/update-dependencies.py
+```
+
+## Documentation
+**TODO: describe**
+
+Update the following wiki pages:
+
+* [Licenses dependencies](Licenses-dependencies.md)
+* [Dependencies Fedora Core](Dependencies-Fedora-Core.md)
+* [Dependencies Mac OS X](Dependencies-Mac-OS-X.md)
+* [Dependencies Ubuntu](Dependencies---Ubuntu.md)
+* [Dependencies Windows](Dependencies-Windows.md)
+
+## Alternatives
+**TODO: describe**

docs/sources/developer/Codereview.md

@@ -0,0 +1,58 @@
+## Code Review
+
+All code submitted into the plaso project goes through code review. We use the GitHub codereview process, which while not perfect, is good enough for our purposes.
+
+One helpful hint is while you have a code in code review monitor the development mailing list for large changes or new dependencies that may potentially affect your code. Such changes may include code re-factors that change plugin interface while you have a plugin in review. These should be rare but they do happen every now and then.
+
+### Rationale
+
+To keep the code base maintainable and readable all code is developed using a similar coding style. See the [style guide](Style-guide). This makes the code easier to maintain and understand.
+
+The purpose of the code review is to ensure that:
+
+ * at least two eyes looked over the code in hopes of finding potential bugs or errors (before they become bugs and errors). This also improves the overall code quality.
+ * make sure the code adheres to the style guide (we do have a linter but that is not perfect).
+ * review design decisions and if needed assist with making the code more optimal or error tolerant.
+
+**The short version:**
+
+*don't be intimidated.*
+
+**The longer version:**
+
+One language is not the same as another, you might are fluent in C or Perl that does not mean the same for Python. You might have just started programming while others have been doing this for years. Our challenge is having a code base that is accessible and sufficiently uniform to most of you.
+
+Also don't be intimidated by rewrites/refactors, which often feels the code base is changing under your feet. We have to make sure the code base is maintainable and a necessary evil there is to regular reshape and clean up things to get new features in.
+
+We continuously try to improve the code base, including making things and easier and quicker to write which sometimes means that the way you just learned might already superseded by another. We try to keep the documentation up to date but this will sometimes be after you ran into an issue.
+
+First time contributors may come across the fact that the code review process actually takes quite a long time, with lots of back and forth comments. You may think that you are wasting the core developers time, but rest assured you are not. We look at this as an investment of building up good solid code contributors. We would like to make sure our contributors understand the code and the style guide and will make suggestions to the contributor to fix what we think needs improving. Despite spending potentially more time to begin with to get code submitted into the project we believe this investment in code review will result in better code submissions and increased proficiency of the contributor.
+
+Therefore we would like to ask people to hang on, to get through the code review process and try to learn something while going through it. Rest assured, it will get easier next time and even easier the time after that, and before you know it you can contribute code to the project with little to no comments.
+
+And if things are unclear, don't hesitate to ask. The developer mailing list is: log2timeline-dev@googlegroups.com
+
+### Why not use reviewable.io?
+
+We have looked at [reviewable.io](https://reviewable.io) and our current assessment is that it looks very nice but does not make for a very functional User Interface/Experience. It also convolutes the git commit history.
+
+#### Referencing github issues
+
+If your changes relate to a specific [github issue](https://github.com/log2timeline/plaso/issues) add the issue number as following:
+
+```
+Added serializers profiler #120
+```
+
+Where the "#120" is a reference to issue number 120.
+
+#### Updating the code review
+
+During the code review process you'll be asked to change few things, that is the reviewer will add comments. Please follow the following guideline during the code review process:
+
+* Answer **ALL** comments made in the code review, even if it is only an ACK or "Done".
+  * It is also necessary to publish the comments, otherwise the reviewer doesn't see the answers.
+  * On the codereview site hit "m" for "Publish+Mail Comments" so that the review gets updated alongside the newly updated code.
+* Make the necessary changes to the code, as suggested by the reviewer.
+
+The update process continues until the reviewer thinks the code is good enough to be submitted into the project. 

docs/sources/developer/Developers-Guide.md

@@ -0,0 +1,88 @@
+## Developer Guide
+
+* [Setting up and maintaining your development environment](Developers-Guide#setting-up-and-maintaining-your-development-environment)
+* [Getting Started](Developers-Guide#getting-started)
+* [Design](Developers-Guide#design)
+* [Roadmap](Developers-Guide#roadmap)
+* [Contributing Code](Developers-Guide#contributing-code)
+
+## Setting up and maintaining your development environment
+
+The first challenge you will encounter is setting up and maintaining your development environment.
+
+**Note that plaso and some dependencies are currently actively under development keeping up with the development release is not for "the faint of heart".**
+
+* [Running the development release on Ubuntu Linux](Development-release-Ubuntu.md)
+  * [Building and installing dependencies on Ubuntu Linux](Dependencies---Ubuntu.md)
+* [Running the development release on Fedora Core Linux](Development-release-Fedora-Core.md)
+  * [Building and installing dependencies on Fedora Core Linux](Dependencies-Fedora-Core.md)
+* [Running the development release on Mac OS X](Development-release-MacOS.md)
+  * [Building and installing dependencies on Mac OS X](Dependencies-Mac-OS-X.md)
+* [Running the development release on Windows](Development-release-Windows.md)
+  * [Building and installing dependencies on Windows](Dependencies-Windows.md)
+
+## Getting Started
+
+Once you've set up your development environment we recommend start simple:
+
+* [How to write a parser](How-to-write-a-parser-plugin.md)
+* [How to write a parser plugin](How-to-write-a-parser-plugin.md)
+* [How to write an analysis plugin](How-to-write-an-analysis-plugin.md)
+* [How to write an output module](How-to-write-an-output-module.md)
+
+## Design
+Overview of the general architecture of plaso:
+
+* [Architecture](Internals.md)
+* [Source code and API documentation](http://plaso-api.readthedocs.org/en/latest/modules.html)
+
+## Roadmap
+
+A high level roadmap can be found [here](Releases-and-roadmap.md). Individual features are tracked as a github issue and labeled as "enhancement". A list of features can be found [here](https://github.com/log2timeline/plaso/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement).
+
+## Contributing Code
+
+Want to add a parser to plaso and you are ready to go? Start by checking [here](https://github.com/log2timeline/plaso/issues?q=is%3Aopen+is%3Aissue+label%3Aenhancement) if someone is already working on it. If you don't see anything there you can just go ahead and [create an issue on the github site](https://github.com/log2timeline/plaso/issues) and mark it as "enhancement". Assign the issue to yourself so that we can keep track on who is working on what.
+
+If you cannot program and still have a great idea for a feature please go ahead and create an issue and leave it unassigned, note that the priority will be who ever wants to work on it. Or consider this the idea opportunity to learn yourself Python programming.
+
+Before you start writing the code, please review the following:
+
+* [Style guide](Style-guide.md). All code submitted to the project needs to follow this style guide.
+* [Code review](Codereview.md). All code that is submitted into the project needs to be reviewed by at least one other person.
+* [Adding a new dependency](Adding-a-new-dependency.md). If your code requires adding a new dependency please check out these instructions.
+
+### Before you submit your first code review
+
+1. Join the development mailing list: [log2timeline-dev@googlegroups.com](https://groups.google.com/forum/?fromgroups#!forum/log2timeline-dev.md), we recommend using the same account as step 1
+2. Install the required development tools like pylint, python-mock, sphinx
+3. Make sure to run all the tests inside the plaso and the dfVFS codebase, and that they successfully complete on your development system
+4. Make sure your development system is set up correctly so that you can develop and test correctly.
+5. Make sure your email address and name are correctly set in git e.g.:
+```
+git config --global user.name "Full Name"
+git config --global user.email name@example.com
+git config --global push.default matching
+```
+
+Use `git config -l` to determine the current configuration.
+
+
+### Core features changes
+
+Sometimes you need to make some change to the core of the plaso codebase. In those cases we ask that contributors first create a short design doc explaining the rationale behind the change. The design doc needs to contain:
+
+1. Describe the problem you are facing
+2. List the objectives of this change
+3. Mention what is in scope and what's not
+4. Describe the solution/proposal
+
+The preferred way of creating these design docs is to use Google Docs and send the link to the development mailing list so that it can be discussed further **before** starting to implement the code.
+
+The current design docs are [stored here](https://drive.google.com/folderview?id=0B3fBvzttpiiSQW16cFhNTUtXVGM&usp=sharing). You may not have access to that folder, so you may need to request access to it.
+
+### Tests
+
+Tests are part of a maintainable code base. Code without sufficient test is very likely to be broken by a large rewrite/refactor.
+
+Some guidelines for writing tests: [Style guide - tests](Style-guide#tests)

docs/sources/developer/Development-release-Fedora-Core.md

@@ -0,0 +1,24 @@
+To download the latest version of Plaso you'll need to install the git tools:
+```
+sudo dnf install git
+```
+
+Checkout the plaso source from the git repo:
+```
+git clone https://github.com/log2timeline/plaso.git
+```
+
+To be able to run the plaso [development release](Releases-and-roadmap) on Fedora Core or equivalent you'll have to have installed the [dependencies](Dependencies).
+
+Check if you have all the dependencies installed and have the right minimum version:
+```
+python utils/check_dependencies.py
+```
+
+**Note that some dependencies are actively under development and can be frequently updated, therefore we recommend checking the status of the dependencies regularly.**
+
+## Development tools
+If you intend to do development on plaso you'll also need to install some development tools:
+
+* PyLint 1.7.x
+* Python Mock

docs/sources/developer/Development-release-MacOS.md

@@ -0,0 +1,28 @@
+To download the latest version of Plaso you'll need to install the git tools: http://git-scm.com/downloads
+
+Checkout the plaso source from the git repo:
+```
+git clone https://github.com/log2timeline/plaso.git
+```
+
+To be able to run the plaso [development release](Releases-and-roadmap) on MacOS you'll have to have installed the [dependencies](Dependencies).
+
+Check if you have all the dependencies installed and have the right minimum version:
+```
+./utils/check_dependencies.py
+```
+
+**Note that some dependencies are actively under development and can be frequently updated, therefore we recommend checking the status of the dependencies regularly.**
+
+If check_dependencies.py keeps indicating it detected an out of date version check if the following directory might still contain an older version:
+```
+/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/
+```
+
+Apple also ships version 2.0.1 of pyparsing under ```/System/Library/Frameworks/Python.framework``` which is loaded first, even if you have a newer pyparsing installed. You can work around this by specifying the PYTHONPATH when you run one of the command line tools - try ```PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH ./tools/log2timeline.py --help``` if you're having problems.
+
+## Development tools
+If you intend to do development on plaso you'll also need to install some development tools:
+
+* PyLint 1.7.x
+* Python Mock