Python Shell HTML JavaScript GAP CSS
Latest commit 989e801 Feb 17, 2017 @ekcs ekcs Exclude atomic rule from dependency graph
Dependency graph used by agnostic excludes positive literals
of type compile.Literal. But when a policy is deleted, the facts
in the policy are deleted as atomic rules of type compile.Rule. As
a result, some nodes in the dependency graph are prematurely deleted
because the deleting of facts (type compile.Rule) via policy delete
decreased ref counters that had not been correspondingly increased
by the adding of those same facts (type compile.Literal). When a
dependency graph is thus corrupted, congress gives Internal Server
Error on all future attempts to add/delete/sync rules.

This patch fixes the problem by having agnostic treat an atomic rule
(type compile.Rule) the same way it treats an atom
(type compile.Literal) in the dependency graph -- ignore both.

Closes-Bug: 1662809

Change-Id: I8080cbb7c375d90259f7b8a2a62d714ebe4aee5f
Permalink
Failed to load latest commit information.
antlr3runtime Fix for broken antlr3 in stand-alone install Mar 24, 2016
bin Fix relative import path in source tree Jul 8, 2014
congress Exclude atomic rule from dependency graph Feb 17, 2017
congress_dashboard Merge "Display id column in PolicRule data in horizon" Dec 24, 2016
congress_tempest_tests Refactor policy synchronizer Jan 12, 2017
contrib/nova Implement horizon plugin model Aug 10, 2016
devstack Lock table and sync rule before each rule insert Jan 19, 2017
doc/source Merge "Update standalone installation section" Feb 2, 2017
etc Set HTTPProxyTOWSGI middleware in front of congress. Oct 10, 2016
examples Remove support for policy snapshot file Apr 20, 2015
releasenotes Update reno for stable/ocata Feb 3, 2017
scripts Adds output_policy_command.py in scripts directory Jan 7, 2017
thirdparty Remove antlr3 files except Python runtime Jan 26, 2017
tools remove new_arch in tox Sep 5, 2016
.coveragerc Update .coveragerc after the removal of openstack directory Oct 17, 2016
.gitignore Update .gitignore with .idea May 24, 2015
.gitreview Update .gitreview file to reflect repo rename Apr 18, 2015
.mailmap Restructure to follow OpenStack cookiecutter template Jan 7, 2014
.testr.conf remove new_arch in tox Sep 5, 2016
CONTRIBUTING.rst Fix few typos Sep 18, 2015
HACKING.rst Fix few typos Sep 18, 2015
LICENSE Restructure to follow OpenStack cookiecutter template Jan 7, 2014
Makefile Remove antlr3 files except Python runtime Jan 26, 2017
README.rst Merge "Update standalone installation section" Feb 2, 2017
babel.cfg Restructure to follow OpenStack cookiecutter template Jan 7, 2014
bindep.txt local HA tests Nov 2, 2016
future-features.txt Fixed some misspellings Jul 1, 2015
requirements.txt Updated from global requirements Feb 10, 2017
run_tests.sh Cleanup unused Oslo Incubator code and references Mar 18, 2016
setup.cfg Remove a py34 environment from tox Feb 10, 2017
setup.py Move antlr version choice out of setup.py Dec 4, 2015
test-requirements.txt Updated from global requirements Feb 10, 2017
thirdparty-requirements.txt Fix spelling typos and one underline Feb 3, 2016
tox.ini Remove a py34 environment from tox Feb 10, 2017

README.rst

Team and repository tags

Congress Introduction and Installation

1. What is Congress

Congress is an open policy framework for the cloud. With Congress, a cloud operator can declare, monitor, enforce, and audit "policy" in a heterogeneous cloud environment. Congress gets inputs from a cloud's various cloud services; for example in OpenStack, Congress fetches information about VMs from Nova, and network state from Neutron, etc. Congress then feeds input data from those services into its policy engine where Congress verifies that the cloud's actual state abides by the cloud operator's policies. Congress is designed to work with any policy and any cloud service.

2. Why is Policy Important

The cloud is a collection of autonomous services that constantly change the state of the cloud, and it can be challenging for the cloud operator to know whether the cloud is even configured correctly. For example,

  • The services are often independent from each other and do not support transactional consistency across services, so a cloud management system can change one service (create a VM) without also making a necessary change to another service (attach the VM to a network). This can lead to incorrect behavior.
  • Other times, we have seen a cloud operator allocate cloud resources and then forget to clean them up when the resources are no longer in use, effectively leaving garbage around the system and wasting resources.
  • The desired cloud state can also change over time. For example, if a security vulnerability is discovered in Linux version X, then all machines with version X that were ok in the past are now in an undesirable state. A version number policy would detect all the machines in that undesirable state. This is a trivial example, but the more complex the policy, the more helpful a policy system becomes.

Congress's job is to help people manage that plethora of state across all cloud services with a succinct policy language.

3. Using Congress

Setting up Congress involves writing policies and configuring Congress to fetch input data from the cloud services. The cloud operator writes policy in the Congress policy language, which receives input from the cloud services in the form of tables. The language itself resembles datalog. For more detail about the policy language and data format see :ref:`Policy <policy>`.

To add a service as an input data source, the cloud operator configures a Congress "driver," and the driver queries the service. Congress already has drivers for several types of service, but if a cloud operator needs to use an unsupported service, she can write a new driver without much effort and probably contribute the driver to the Congress project so that no one else needs to write the same driver.

Finally, when using Congress, the cloud operator must choose what Congress should do with the policy it has been given:

  • monitoring: detect violations of policy and provide a list of those violations
  • proactive enforcement: prevent violations before they happen (functionality that requires other services to consult with Congress before making changes)
  • reactive enforcement: correct violations after they happen (a manual process that Congress tries to simplify)

In the future, Congress will also help the cloud operator audit policy (analyze the history of policy and policy violations).

Congress is free software and is licensed with Apache.

  • Free software: Apache license

4. Installing Congress

There are 2 ways to install Congress.

  • As part of DevStack. Get Congress running alongside other OpenStack services like Nova and Neutron, all on a single machine. This is a great way to try out Congress for the first time.
  • Separate install. Get Congress running alongside an existing OpenStack deployment

4.1 Devstack-install

For integrating Congress with DevStack:

  1. Download DevStack
$ git clone https://git.openstack.org/openstack-dev/devstack.git
$ cd devstack
  1. Configure DevStack to use Congress and any other service you want. To do that, modify the local.conf file (inside the DevStack directory). Here is what our file looks like:
[[local|localrc]]

enable_plugin congress http://git.openstack.org/openstack/congress
enable_plugin heat http://git.openstack.org/openstack/heat
enable_plugin aodh http://git.openstack.org/openstack/aodh
enable_plugin ceilometer http://git.openstack.org/openstack/ceilometer
enable_service s-proxy s-object s-container s-account
  1. Run stack.sh. The default configuration expects the passwords to be 'password' without the quotes
$ ./stack.sh

4.2 Separate install

Install the following software, if you haven't already.

$ sudo apt-get install git gcc python-dev python-antlr3 libxml2 libxslt1-dev libzip-dev build-essential libssl-dev libffi-dev
$ sudo apt install python-setuptools
$ sudo pip install --upgrade pip virtualenv pbr tox

Clone Congress

$ git clone https://github.com/openstack/congress.git
$ cd congress

Install requirements

$ sudo pip install .

Install Source code

$ sudo python setup.py install

Configure Congress (Assume you put config files in /etc/congress)

$ sudo mkdir -p /etc/congress
$ sudo mkdir -p /etc/congress/snapshot
$ sudo cp etc/api-paste.ini /etc/congress
$ sudo cp etc/policy.json /etc/congress

Generate a configuration file as outlined in the Configuration Options section of the :ref:`Deployment <deployment>` document. Note: you may have to run the command with sudo.

There are several sections in the congress/etc/congress.conf.sample file you may want to change:

  • [DEFAULT] Section
    • drivers
    • auth_strategy
  • "From oslo.log" Section
    • log_file
    • log_dir (remember to create the directory)
  • [database] Section
    • connection

Add drivers:

drivers = congress.datasources.neutronv2_driver.NeutronV2Driver,congress.datasources.glancev2_driver.GlanceV2Driver,congress.datasources.nova_driver.NovaDriver,congress.datasources.keystone_driver.KeystoneDriver,congress.datasources.ceilometer_driver.CeilometerDriver,congress.datasources.cinder_driver.CinderDriver,congress.datasources.swift_driver.SwiftDriver,congress.datasources.plexxi_driver.PlexxiDriver,congress.datasources.vCenter_driver.VCenterDriver,congress.datasources.murano_driver.MuranoDriver,congress.datasources.ironic_driver.IronicDriver

The default auth_strategy is keystone. To set Congress to use no authorization strategy:

auth_strategy = noauth

If you use noauth, you might want to delete or comment out the [keystone_authtoken] section.

Set the database connection string in the [database] section (adapt MySQL root password):

connection = mysql+pymysql://root:password@127.0.0.1/congress?charset=utf8

To use RabbitMQ with Congress, set the transport_url in the "From oslo.messaging" section according to your setup:

transport_url = rabbit://$RABBIT_USERID:$RABBIT_PASSWORD@$RABBIT_HOST:5672

A bare-bones congress.conf is as follows:

[DEFAULT]
auth_strategy = noauth
drivers = congress.datasources.neutronv2_driver.NeutronV2Driver,congress.datasources.glancev2_driver.GlanceV2Driver,congress.datasources.nova_driver.NovaDriver,congress.datasources.keystone_driver.KeystoneDriver,congress.datasources.ceilometer_driver.CeilometerDriver,congress.datasources.cinder_driver.CinderDriver,congress.datasources.swift_driver.SwiftDriver,congress.datasources.plexxi_driver.PlexxiDriver,congress.datasources.vCenter_driver.VCenterDriver,congress.datasources.murano_driver.MuranoDriver,congress.datasources.ironic_driver.IronicDriver
log_file=congress.log
log_dir=/var/log/congress
[database]
connection = mysql+pymysql://root:password@127.0.0.1/congress?charset=utf8

When you are finished editing congress.conf.sample, copy it to the /etc/congress directory.

sudo cp etc/congress.conf.sample /etc/congress/congress.conf

Create database

$ mysql -u root -p
$ mysql> CREATE DATABASE congress;
$ mysql> GRANT ALL PRIVILEGES ON congress.* TO 'congress'@'localhost' IDENTIFIED BY 'CONGRESS_DBPASS';
$ mysql> GRANT ALL PRIVILEGES ON congress.* TO 'congress'@'%' IDENTIFIED BY 'CONGRESS_DBPASS';

Push down schema

$ sudo congress-db-manage --config-file /etc/congress/congress.conf upgrade head
Set up Congress accounts

Use your OpenStack RC file to set and export required environment variables: OS_USERNAME, OS_PASSWORD, OS_PROJECT_NAME, OS_TENANT_NAME, OS_AUTH_URL.

(Adapt parameters according to your environment)

$ ADMIN_ROLE=$(openstack role list | awk "/ admin / { print \$2 }")
$ SERVICE_TENANT=$(openstack project list | awk "/ service / { print \$2 }")
$ CONGRESS_USER=$(openstack user create --password password --project service --email "congress@example.com" congress | awk "/ id / {print \$4 }")
$ openstack role add $ADMIN_ROLE --user $CONGRESS_USER --project  $SERVICE_TENANT
$ CONGRESS_SERVICE=$(openstack service create policy --name congress --description "Congress Service" | awk "/ id / { print \$4 }")
Create the Congress Service Endpoint
Endpoint creation differs based upon the Identity version. Please see the endpoint documentation for details.
Identity v2:
$ openstack endpoint create $CONGRESS_SERVICE --region RegionOne --publicurl http://127.0.0.1:1789/  --adminurl http://127.0.0.1:1789/ --internalurl http://127.0.0.1:1789/
Identity v3:
$ openstack endpoint create --region $OS_REGION_NAME  $CONGRESS_SERVICE public http://$SERVICE_HOST:1789
$ openstack endpoint create --region $OS_REGION_NAME  $CONGRESS_SERVICE admin http://$SERVICE_HOST:1789
$ openstack endpoint create --region $OS_REGION_NAME  $CONGRESS_SERVICE internal http://$SERVICE_HOST:1789
Start Congress
The default behavior is to start the Congress API, Policy Engine, and Datasource in a single node. For HAHT deployment options, please see the :ref:`HA Overview <ha_overview>` document.
$ sudo /usr/local/bin/congress-server --debug
Install the Congress Client
The command line interface (CLI) for Congress resides in a project called python-congressclient. Follow the installation instructions on the GitHub page.
Configure datasource drivers
For this you must have the Congress CLI installed. Run this command for every service that Congress will poll for data. Please note that the service name $SERVICE should match the ID of the datasource driver, e.g. "neutronv2" for Neutron and "glancev2" for Glance; $OS_USERNAME, $OS_TENANT_NAME, $OS_PASSWORD and $SERVICE_HOST are used to configure the related datasource driver so that congress knows how to talk with the service.
$ openstack congress datasource create $SERVICE $"SERVICE" \
  --config username=$OS_USERNAME \
  --config tenant_name=$OS_TENANT_NAME
  --config password=$OS_PASSWORD
  --config auth_url=http://$SERVICE_HOST:5000/v2.0
Install the Congress Dashboard in Horizon
Follow the instructions in the README file located in the congress/congress_dashboard directory. Note: After you install the Congress Dashboard and restart apache, the OpenStack Dashboard may throw a "You have offline compression enabled..." error, follow the instructions in the error message. You may have to:
$ cd /opt/stack/horizon
$ python manage.py compress
$ sudo service apache2 restart
Read the HTML documentation
Install python-sphinx and the oslosphinx extension if missing and build the docs. After building, open congress/doc/html/index.html in a browser.
$ sudo pip install sphinx
$ sudo pip install oslosphinx
$ make docs
Test Using the Congress CLI

If you are not familiar with using the OpenStack command-line clients, please read the OpenStack documentation before proceeding.

Once you have set up or obtained credentials to use the OpenStack command-line clients, you may begin testing Congress. During installation a number of policies are created.

To view policies: $ openstack congress policy list

To view installed datasources: $ openstack congress datasource list

To list available commands: $ openstack congress --help

4.3 Unit Tests

Run unit tests in the Congress directory

$ tox -epy27

In order to break into the debugger from a unit test we need to insert a break point to the code:

import pdb; pdb.set_trace()

Then run tox with the debug environment as one of the following:

tox -e debug
tox -e debug test_file_name.TestClass.test_name

For more information see the oslotest documentation.

4.4 Upgrade

Here are the instructions for upgrading to a new release of the Congress server.

  1. Stop the Congress server.
  2. Update the Congress git repo
$ cd /path/to/congress
$ git fetch origin

3. Checkout the release you are interested in, say Mitaka. Note that this step will not succeed if you have any uncommitted changes in the repo.

$ git checkout origin/stable/mitaka

If you have changes committed locally that are not merged into the public repository, you now need to cherry-pick those changes onto the new branch.

  1. Install dependencies
$ sudo pip install
  1. Install source code
$ sudo python setup.py install
  1. Migrate the database schema
$ sudo congress-db-manage --config-file /etc/congress/congress.conf upgrade head
  1. (optional) Check if the configuration options you are currently using are still supported and whether there are any new configuration options you would like to use. To see the current list of configuration options, use the following command, which will create a sample configuration file in etc/congress.conf.sample for you to examine.
$ tox -egenconfig
  1. Restart Congress, e.g.
$ sudo /usr/local/bin/congress-server --debug