MOX Messaging Service and Actual State Database
- System requirements
- Getting started
This project contains an implementation of the OIO object model, used as a standard for data exchange by the Danish government, for use with a MOX messaging queue.
You can find the current MOX specification here:
As an example, you can find the Organisation hierarchy here:
In each installation of MOX, it is possible to only enable some of the hierarchies, but we provide the following four OIO hierarchies by default:
This README file is part of our documentation, and an HTML version can be obtained by the following command in a command prompt:
$ make -C doc
Note that this requires Sphinx to be installed — on Ubuntu or Debian, this can be done with the following command:
$ sudo apt install python-sphinx
If you're reading this on GitHub or ReadTheDocs, you're probably seeing the HTML rendering. The official location for this documentation is:
Please note that as a convention, all shell commands have been
prefixed with a dollar-sign, or
$, representing a prompt. You
should exclude this when entering the command in your terminal.
This is a technical guide. You are not expected to have a profound knowledge of the system as such, but you do have to know your way in a Bash prompt — you should be able to change the Apache configuration and e.g. disable or change the SSL certificate on your own.
LoRA currently supports Ubuntu 16.04 (Xenial). We recommend running it on a VM with the following allocation:
|Minimal||1 core||2 GB||15 GB||any (SSD or HD)|
|Test & development||2 cores||4 GB||30 GB||SSD (recommended)|
|Production||4 cores||8 GB||60 GB||SSD|
You should initially provision all the storage space you expect to use, as adjusting it is somewhat cumbersome. By comparison, increasing or decreasing CPU and memory is trivial.
To install the OIO REST API, run
$ sudo apt-get install git $ git clone https://github.com/magenta-aps/mox $ cd mox $ ./install.sh
Using the built-in installer should be considered a "developer installation". Please note that it will not work on a machine with less than 1GB of RAM.
In the current state of the installer, we recommend using it only on a newly installed system, preferrably in a dedicated container or VM. It is our intention to provide more flexible installation options in the near future.
The default location of the log directory is:
The following log files are created:
- Audit log: /var/log/mox/audit.log
- OIO REST HTTP access log: /var/log/mox/oio_access.log
- OIO REST HTTP error log: /var/log/mox/oio_error.log
Additionally, a directory for file uploads is created:
# settings.py FILE_UPLOAD_FOLDER = getenv('FILE_UPLOAD_FOLDER', '/var/mox')
The oio rest api is installed as a service, for more information, see the oio_rest.service.
By default the oio_rest service can be reached as follows:
In a production environment, it is recommended to bind the oio_rest service to a unix socket, then expose the socket using a HTTP proxy of your own choosing.
(Recommended: Nginx or Apache)
More on how to configure in the advanced configuration document.
Document is currently being written
Most configurable parameters of oio_rest can be injected with environment variables, alternatively you may set the parameters explicitly in the "settings.py" file.
(Please see $ROOT/oio_rest/settings.py)
As mentioned, most parameters are accessible. If they are NOT set by you, we have provided sensible fallback values.
# settings.py # DB (Postgres) settings DATABASE = getenv('DB_NAME', 'mox') DB_USER = getenv('DB_USER', 'mox') DB_PASSWORD = getenv('DB_PASS', 'mox')
The oio rest api is served using the wsgi server gunicorn. The gunicorn server can be configured through the "guniconfig.py" file.
(Please see $ROOT/oio_rest/guniconfig.py)
Similar to the oio rest settings file, gunicorn can be configured using environment variables:
# guniconfig.py # Bind address (Can be either TCP or Unix socket) bind = env("GUNICORN_BIND_ADDRESS", '127.0.0.1:8080')
On a high level the MOX actual state database consists of three server processes and several agents joining them together.
- Database server providing the storage of the bi-temporal actual state database as well as validation and verification of the basic constraints.
- WSGI server for the oio rest api. Ideally used with a frontend HTTP proxy in production.
- AMQP message broker providing interprocess communication between the various components.
Within the context of the Mox Messaging Service, agents are small pieces of software which either listen on an AMQP queue and perform operations on the incoming data, or expose certain operations as a web service.
The default installation includes the following agents:
- Web service for exporting actual state contents as Excel spreadsheets.
- Web service for importing data from Excel spreadsheets into the actual state database.
- AMQP agent bridging the REST API.
- AQMP worker agent MoxDocumentDownload & MoxDocumentUpload.
SAML token authentication is enabled by default. This requires that you have access to a SAML Identity Provider (IdP) which provides a Security Token Service (STS). We currently support two types:
- Active Directory Federation Services
In order to use AD FS as the Security Token Service, you first need an endpoint configured in ADFS. You should name this endpoint corresponding to the designated name of the box running LoRA, for example:
As for the attributes to send, select the following:
|LDAP Attribute||Outgoing Claim Type|
|Token-Groups (Unqualified Names)||Group|
Please note that you should configure AD FS to sign, but not encrypt, its assertions.
Then configure the following fields in
||In this case, “
||The name of your ADFS.|
||The URL where your ADFS may be reached.|
You should now be able to test the basic configuration, and extract the signing certificate:
$ cd /path/to/mox $ ./auth.sh --cert-only User: user@domain Password: <enter password here>
Now save the results to a file, e.g.
adfs-cert.pem, and set that
SAML_IDP_CERTIFICATE. You may get an SSL error, in that case,
you should add your certificate authority to the system.
Alternatively, you can pass the
--insecure option to
temporarily bypass the error.
The open source identity provider WSO2 is useful for testing. Download the binary and follow the instructions to run it.
In the folder
wso2/ you can find an example init file for running the
WSO2 Identity Server as a daemon.
To configure a STS, follow the instructions on https://docs.wso2.com/display/IS500/Configuring+the+Identity+Server+to+Issue+Security+Tokens (skip the part about Holder of Key).
Restart the WSO2 server! The STS endpoint simply did not work until I restarted the WSO2 server.
This is for testing with the WSO2 Identity Server as described above - we assume that this is not the configuration which the municipalities want to use in a production setting.
Log in to the IDP with the credentials provided. The IDP could, e.g., be located at https://moxtest.magenta-aps.dk:9443/.
To create a new user, enter the "Configure" tab and select "Users and roles". Enter the user's first name, last name and email address.
Important: In the URL field, enter the user's (OIO) UUID. The URL field is currently used to map between the IDP and the OIO's user concept. If the UUID is not specified, it will not be possible to authorize users correctly, nor will it be possible to make any changes to the database.
The default IdP entity ID is called "localhost". If your IdP has a different entity ID, you must change the SAML_IDP_ENTITY_ID setting to reflect your IdP's entity ID.
For testing purposes, WSO2's IdP public certificate file is included in the distribution.
When configuring the REST API to use your IdP, you must specify your IdP's public certificate file by setting in settings.py:
In settings.py, SAML authentication can be turned off by setting:
USE_SAML_AUTHENTICATION = False
The OIO REST service provides a convenience method for requesting a SAML token in the correct base64-encoded gzipped format for use with the API.
Visit the following URL of the OIO REST server:
Alternatively, you can run the following command locally on the server:
$ ./auth.sh -u <username> -p
You will be presented with a form with a username/password field. Optionally, you can specify the STS address to use. This will request a token from the STS service using the given username and password. It will return the value that should be used for the HTTP "Authorization" header. If it fails due to invalid username/password, an error message will be returned.
This value can then be included in the HTTP "Authorization" header, like the following:
Authorization: <output of get-token>
This section only applies covers using the WSO2 IdP.
Although the Java MOX agent does this automatically, it can be useful to request a SAML token manually, for testing purposes.
To request a SAML token, it is useful to use SoapUI.
Download SoapUI and import the project
Navigate to and double-click on:
"sts" -> "wso2carbon-stsSoap11Binding" -> "Issue token - SAML 2.0"
Note: The value of
<a:Address> element in
<wsp:AppliesTo> must match your
SAML_MOX_ENTITY_ID setting. Change as needed.
The project assumes you are running the IdP server on https://localhost:9443/ (the default).
Execute the SOAP request. You can copy the response by clicking on the
"Raw" tab in the right side of the window and then selecting all, and
copying to the clipboard. Paste the response, making sure that the
original whitespace/indentation is preserved. Remove all elements/text
<saml2:Assertion>..</saml2:Assertion> tag. Save to a
file, e.g. /my/saml/assertion.xml.
After requesting a SAML token, to make a REST request using the SAML token, you need to pass in an HTTP Authorization header of a specific format:
Authorization: saml-gzipped <base64-encoded gzip-compressed SAML assertion>
A script has been included to generate this HTTP header from a SAML token
XML file. This file must only contain the
To run it:
$ python oio_rest/oio_rest/utils/encode_token.py /my/saml/assertion.xml
The output of this script can be used in a curl request by adding the parameter -H, e.g.:
$ curl -H "Authorization saml-gzipped eJy9V1................." ...
to the curl request.
Alternately, if using bash shell:
$ curl -H "$(python oio_rest/oio_rest/utils/encode_token.py" /my/saml/assertion.xml) ...
To run the API for testing or development purposes, run:
Then, go to
http://localhost:5000/site-map to see a map of all available
URLs, assuming you're running this on your local machine.
The install.sh script creates an Apache VirtualHost for oio rest and MoxDocumentUpload.
To run the OIO Rest Mox Agent (the one listening for messages and relaying them onwards to the REST interface), run:
NOTE:** You can start the agent in the background by running:
$ sudo service moxrestfrontend start
To test sending messages through the agent, run:
NOTE: The install script does not set up an IDP for SAML authentication, which is enabled by default. If you need to test without SAML authentication, you will need to turn it off as described below.
To request a token for the username from the IdP and output it in base64-encoded gzipped format, run:
$ ./auth.sh -u <username> -p
Insert your username in the command argument. You will be prompted to enter a password.
If SAML authentication is turned on (i.e., if the parameter
True), the IDP must be configured correctly — see the corresponding
sections below for instruction on how to do this.
Make sure the parameter
oio_rest/oio_rest/settings.py is False.
Make sure the parameter
agents/MoxRestFrontend/moxrestfrontend.conf is set to
Start the (AMQP) MOX REST frontend agent:
$ sudo service moxrestfrontend start
Start the REST API:
Run the tests:
This should give you a lot of output like this:
Deleting bruger, uuid: 1e874f85-07e5-40e5-81ed-42f21fc3fc9e Getting authtoken 127.0.0.1 - - [27/Apr/2016 15:55:09] "DELETE /organisation/bruger/1e874f85-07e5-40e5-81ed-42f21fc3fc9e HTTP/1.1" 200 - Delete succeeded
Note: Currently, some of the tests will give the notice: "Result differs from the expected". This is due to a bug in the tests, i.e. you should not worry about this — if you see output as described above, the system is working.
Additionally, OIO Rest has its own unit test suite:
$ cd oio_rest $ python setup.py test
$ cd oio_rest $ ./run_tests.sh
The latter of which will automatically generate a virtual environment, and run the tests in it.
For more advanced test or production setup, please study the rest of this README and follow your organization's best practices.
The MOX messaging queue, including the ActualState database, as found
in this project is free software. You are entitled to use, study,
modify and share it under the provisions of Version 2.0 of the
Mozilla Public License as
specified in the