Skip to content

Commit

Permalink
Merge branch 'release-1.5.0'
Browse files Browse the repository at this point in the history
  • Loading branch information
gregcorbett committed Sep 28, 2017
2 parents 5f9bae6 + eb29573 commit f1b2d10
Show file tree
Hide file tree
Showing 29 changed files with 1,353 additions and 724 deletions.
10 changes: 7 additions & 3 deletions .travis.yml
Expand Up @@ -2,10 +2,10 @@ language: python
python:
- "2.6"
- "2.7"
- "3.6"
- "nightly"
matrix:
allow_failures:
- python: "3.6"
- python: "nightly"
fast_finish: true

# use the mysql service
Expand All @@ -16,6 +16,8 @@ services:
sudo: false
# Cache the dependencies installed by pip
cache: pip
# Avoid pip log from affecting cache
before_cache: rm -fv ~/.cache/pip/log/debug.log

install:
- pip install -r requirements.txt
Expand All @@ -26,9 +28,11 @@ before_script:
# create an empty mysql database
- mysql -u root -e "create database apel_rest"
- mysql -u root apel_rest < schemas/10-cloud.sql
# partition the database, at least to make sure the syntax is correct
- mysql -u root apel_rest < schemas/20-cloud-extra.sql
- export PYTHONPATH=$PYTHONPATH:`pwd -P`

# Command to run tests
script: coverage run --source='.' manage.py test
script: coverage run --branch --source='.' manage.py test

after_success: coveralls
4 changes: 2 additions & 2 deletions Dockerfile
@@ -1,9 +1,9 @@
FROM centos:6
FROM centos:7

MAINTAINER APEL Administrator <apel-admins@stfc.ac.uk>

# Add EPEL repo so we can get pip
RUN rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
RUN rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-10.noarch.rpm

# Add IGTF trust repo, so it can be installed
RUN touch /etc/yum.repos.d/EGI-trustanchors.repo
Expand Down
59 changes: 37 additions & 22 deletions README.md
Expand Up @@ -9,31 +9,38 @@ APEL Cloud Accounting can account for the usage of OpenNebula and OpenStack inst

The collectors produce "Usage Records" in the APEL-Cloud v0.2 or v0.4 message formats. Information about these format can be found [here](https://wiki.egi.eu/wiki/Federated_Cloud_Accounting#Documentation).

These records need to be sent as POST requests to the REST endpoint `.../api/v1/cloud/record`, where `...` is the machine hosting the docker image. A POST request requires an X.509 certificate to authenticate the request. The hostname, which should be the same as the common name (CN) contained in the X.509 certificate, must be listed as a provider [here](http://indigo.cloud.plgrid.pl/cmdb/service/list) for the request to be authorized.
These records need to be sent as POST requests to the REST endpoint `.../api/v1/cloud/record`, where `...` is the machine hosting the docker image. A POST request requires an X.509 certificate to authenticate the request. The hostname, which should be the same as the common name (CN) contained in the X.509 certificate, must be listed as a provider [here](http://indigo.cloud.plgrid.pl/cmdb/service/list) or be given special access via the `ALLOWED_TO_POST` enviroment variable for the request to be authorized. For details of how to be added to the CMDB service list, please refer to the [CMDB documentation](https://github.com/indigo-dc/cmdb/blob/master/README.md).

Accepted records are summarised twice daily. These summaries can be accessed with a GET request to `.../api/v1/cloud/record/summary`. Summaries can be filtered using `key=value` pairs. See [Supported key=value pairs](doc/user.md#supported-keyvalue-pairs) for a list of valid supported `key=value` pairs. A GET request requires an IAM access token be included in the request. This token is then sent to the IAM to authenticate the ID of the service requesting access to the summary. This ID needs to be in `ALLOWED_FOR_GET` in `yaml/apel_rest_interface.env` for access to be authorized. See [Authorize new WP5 components to view Summaries](doc/admin.md#authorize-new-wp5-components-to-view-summaries) for instructions on adding service to `ALLOWED_FOR_GET`
Accepted records are summarised twice daily. These summaries can be accessed with a GET request to `.../api/v1/cloud/record/summary`. Summaries can be filtered using `key=value` pairs. See [Supported key=value pairs](doc/user.md#supported-keyvalue-pairs) for a list of valid supported `key=value` pairs. A GET request requires an IAM access token be included in the request. This token is then sent to the IAM to authenticate the ID of the service requesting access to the summary. This ID needs to be in `ALLOWED_FOR_GET` in `yaml/apel_rest_interface.env` for access to be authorized.

It is currently expected that only the QoS/SLA tool will interact with these summaries.

### Features of Version 1.4.0-1
### Features of Version 1.5.0

- Accept APEL-Cloud v0.2 and v0.4 usage records via POST requests to the REST endpoint `.../api/v1/cloud/record`
- Provide access to summaries via GET requests to REST endpoint `.../api/v1/cloud/record/summary`

## Running the docker image on Centos 7 and Ubuntu 16.04
We recommend using the docker image to run the Accounting server and REST interface. As such, having Docker and docker-compose installed is a prerequisite.
## Running the Docker image on Centos 7 and Ubuntu 16.04
We recommend using the Docker image to run the Accounting server and REST interface. As such, having Docker installed is a prerequisite.

See [Ubuntu 16.04 Instructions](https://docs.docker.com/engine/installation/linux/ubuntulinux/) or [Centos 7 Instructions](https://docs.docker.com/engine/installation/linux/centos/) for details of how to install Docker.
See [Ubuntu 16.04 Instructions](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/) or [Centos 7 Instructions](https://docs.docker.com/engine/installation/linux/docker-ce/centos/) for details of how to install Docker. There is no requirement to undertake the 'Post-installation steps for Linux'.

See [Install Docker Compose](https://docs.docker.com/compose/install/) for details of how to install docker-compose
We also recommend using Docker Compose or Anisble to deploy the containers.
* See [the offical Docker Compose documentation](https://docs.docker.com/compose/install/) for details of how to install docker-compose.
* See [the offical Ansible documentation](http://docs.ansible.com/ansible/latest/intro_installation.html#installation) for details of how to install Ansible.

1. Download the source code for the version you wish to deploy, see [here](https://github.com/indigo-dc/Accounting/releases) for a list of releases and corresponding docker image tag. The source code contains schemas and yaml files needed for deploying via docker.
The instructions below are for both Docker Compose and Ansibe.

2. Register the service as a protected resource with the Indigo Identity Access Management (IAM) service. See [here](doc/admin.md#register-the-service-as-a-protected-resource-with-the-indigo-identity-access-management-iam) for instructions.
1. Download the compressed source code from [here](https://github.com/indigo-dc/Accounting/releases/latest), it does not matter where the source code is downloaded to. As the source code contains schemas and yaml files needed for deploying via docker-compose or ansible, you will need to unzip the source code and then `cd` into the inflated directory.

2. Register the service as a protected resource with the Indigo Identity Access Management (IAM) service. See [this section](doc/admin.md#register-the-service-as-a-protected-resource-with-the-indigo-identity-access-management-iam) for instructions, before proceeding to step 3.

3. Populate the following variables in `yaml/apel_rest_interface.env`
```
DJANGO_SECRET_KEY: The Django server requires its own "secret".
DJANGO_SECRET_KEY: The Django server requires its own "secret". A typical secret key
will be a 50 character long random sequence of alphanumeric
and special characters. This secret will be used for JSON object
signing and other cryptographic functions like creating CSRF keys.
PROVIDERS_URL: Points to the JSON list of Resource Providers
Expand All @@ -42,21 +49,17 @@ See [Install Docker Compose](https://docs.docker.com/compose/install/) for detai
SERVER_IAM_ID: An IAM ID corresponding to this instance, used to validate tokens.
SERVER_IAM_SECRET: An IAM secret corresponding to this instance, used to validate tokens.
ALLOWED_FOR_GET: A (Python) list of IAM service IDs allowed to submit GET requests.
(e.g. ['ac2f23e0-8103-4581-8014-e0e82c486e36'])
ALLOWED_TO_POST: A (Python) list of X.509 HostDNs allowed to submit POST requests,
in addition to those listed by the PROVIDERS_URL.
(e.g. ['/C=XX/O=XX/OU=XX/L=XX/CN=special_host.test'])
BANNED_FROM_POST: A (Python) list of X.509 HostDNs banned from submitting POST requests,
even if they are listed by the PROVIDERS_URL.
(e.g. ['/C=XX/O=XX/OU=XX/L=XX/CN=banned_host.test'])
SERVER_IAM_ID: An IAM ID corresponding to this instance, used to validate tokens.
SERVER_IAM_SECRET: An IAM secret corresponding to this instance, used to validate tokens.
ALLOWED_FOR_GET: A (Python) list of IAM service IDs allowed to submit GET requests.
(e.g. ['ac2f23e0-8103-4581-8014-e0e82c486e36'])
```

4. Populate the following variables in `yaml/mysql.env`
Expand All @@ -72,18 +75,30 @@ See [Install Docker Compose](https://docs.docker.com/compose/install/) for detai

5. `MYSQL_PASSWORD` will also need to be added to the password field `docker/etc/apel/clouddb.cfg` and `docker/etc/mysql/my.cnf`

6. Before the REST interface will start, a certificate needs to be added to the container. This can be done by placing a certificate (`apache.crt`) and key (`apache.key`) under `/etc/httpd/ssl/`. This directory will be mounted into the container by docker-compose.
6. Before the REST interface will start, a certificate needs to be added to the container. This can be done by placing a certificate (`apache.crt`) and key (`apache.key`) under `/etc/httpd/ssl/` on the host machine (be careful to preserve correct `.crt`/`.key` file permissions). `/etc/httpd/ssl/` will be mounted into the container by docker-compose. Note: You may need to create the `/etc/httpd/ssl/` directory.

7. Make `docker/etc/cron.d` owned by `root`. This is required because this directory gets mounted into the container and it needs to be owned by root for cron jobs in the container to run.
7. If you are running Docker as a non-root user, you will need to make `docker/etc/cron.d` owned by `root`. This is because this directory gets mounted into the container and it needs to be owned by `root` for cron jobs in the container to run.
```
chown -R root docker/etc/cron.d
sudo chown -R root docker/etc/cron.d
```

8. Launch the containers. It is recommeded to wait 1 minute in order for each container to configure fully before launching the next
8. Launch the containers.

- With Docker Compose:

It is recommeded to wait 1 minute after launching the database in order for it to configure fully before launching the APEL containers
```
docker-compose -f yaml/docker-compose.yaml up -d --force-recreate apel_mysql
# Wait 1 minute
docker-compose -f yaml/docker-compose.yaml up -d --force-recreate apel_server
docker-compose -f yaml/docker-compose.yaml up -d --force-recreate apel_rest_interface
```

- With Ansible:

First, replace `<working directory>` in `yaml/ansible.yaml` with the absolute file path of the decompressed source code, then run the command below.
```
ansible-playbook yaml/ansible.yml
```

9. Navigate a web browser to `https://<hostname>/api/v1/cloud/record/summary`
2 changes: 1 addition & 1 deletion apel_rest/__init__.py
Expand Up @@ -15,4 +15,4 @@
@author Greg Corbett
"""
__version__ = '1, 4, 0'
__version__ = '1, 5, 0'
3 changes: 2 additions & 1 deletion apel_rest/settings.py
Expand Up @@ -163,7 +163,8 @@

PROVIDERS_URL = 'provider_url'

IAM_URL = 'iam_url'
# List of hostnames of IAMs that can issue access tokens for the REST interface
IAM_HOSTNAME_LIST = ['allowed_iams']
SERVER_IAM_ID = 'server_iam_id'
SERVER_IAM_SECRET = 'server_iam_secret'

Expand Down
2 changes: 1 addition & 1 deletion api/__init__.py
Expand Up @@ -15,4 +15,4 @@
@author Greg Corbett
"""
__version__ = '1, 4, 0'
__version__ = '1, 5, 0'
93 changes: 92 additions & 1 deletion api/tests/__init__.py
@@ -1 +1,92 @@
"""This file allows test to be imported as a python package."""
"""
Copyright (C) 2016 STFC.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
@author Greg Corbett
"""
__version__ = '1, 4, 0'

# Example output from the provider list on the CMDB
PROVIDERS = {'total_rows': 735,
'offset': 695,
'rows': [
{'id': '1',
'key': ['service'],
'value':{
'sitename': 'TEST2',
'provider_id': 'TEST2',
'type': 'cloud'}},
{'id': '2',
'key': ['service'],
'value':{
'sitename': 'TEST',
'provider_id': 'TEST',
'hostname': 'allowed_host.test',
'type': 'cloud'}},
{'id': '3',
'key': ['service'],
'value':{
'sitename': 'TEST',
'provider_id': 'TEST',
'hostname': 'allowed_host2.test',
'type': 'cloud'}}]}

# An example APEL Cloud Message V0.2
MESSAGE = """APEL-cloud-message: v0.2
VMUUID: TestVM1 2013-02-25 17:37:27+00:00
SiteName: CESGA
MachineName: one-2421
LocalUserId: 19
LocalGroupId: 101
GlobalUserName: NULL
FQAN: NULL
Status: completed
StartTime: 1361813847
EndTime: 1361813870
SuspendDuration: NULL
WallDuration: NULL
CpuDuration: NULL
CpuCount: 1
NetworkType: NULL
NetworkInbound: 0
NetworkOutbound: 0
Memory: 1000
Disk: NULL
StorageRecordId: NULL
ImageId: NULL
CloudType: OpenNebula
%%
VMUUID: TestVM1 2015-06-25 17:37:27+00:00
SiteName: CESGA
MachineName: one-2422
LocalUserId: 13
LocalGroupId: 131
GlobalUserName: NULL
FQAN: NULL
Status: completed
StartTime: 1361413847
EndTime: 1361811870
SuspendDuration: NULL
WallDuration: NULL
CpuDuration: NULL
CpuCount: 1
NetworkType: NULL
NetworkInbound: 0
NetworkOutbound: 0
Memory: 1000
Disk: NULL
StorageRecordId: NULL
ImageId: NULL
CloudType: OpenNebula
%%"""
70 changes: 70 additions & 0 deletions api/tests/test_cloud_record_helper.py
@@ -0,0 +1,70 @@
"""This module tests the helper methods of the CloudRecordView class."""

import logging

from django.test import TestCase
from mock import Mock

from api.tests import PROVIDERS
from api.views.CloudRecordView import CloudRecordView


class CloudRecordHelperTest(TestCase):
"""Tests the helper methods of the CloudRecordView class."""

def setUp(self):
"""Prevent logging from appearing in test output."""
logging.disable(logging.CRITICAL)

def test_signer_is_valid(self):
"""
Test the CloudRecordView._signer_is_valid method.
That method determines wether a POST request should be allowed or not.
"""
# mock the external call to the CMDB to retrieve the providers
# used in the _signer_is_valid method we are testing
CloudRecordView._get_provider_list = Mock(return_value=PROVIDERS)
test_cloud_view = CloudRecordView()

# The DN corresponds to a host listed as a provider
allowed_dn = "/C=XX/O=XX/OU=XX/L=XX/CN=allowed_host.test"

# The Host this DN corresponds to will
# be explicitly granted POST rights
extra_dn = "/C=XX/O=XX/OU=XX/L=XX/CN=special_host.test"

# The Host this DN corresponds to will
# have its POST rights explicitly revoked
banned_dn = "/C=XX/O=XX/OU=XX/L=XX/CN=allowed_host2.test"

# The Host this DN corresponds is unknown to the system,
# it should not be granted POST rights
unknown_dn = "/C=XX/O=XX/OU=XX/L=XX/CN=mystery_host.test"

# Grant/Revoke POST fights
with self.settings(ALLOWED_TO_POST=[extra_dn],
BANNED_FROM_POST=[banned_dn]):

# DNs corresponding to hosts listed as a provider should be valid
# i.e have POST rights
self.assertTrue(test_cloud_view._signer_is_valid(allowed_dn))

# DNs corresponding to hosts with explicit access should be valid
# i.e have POST rights
self.assertTrue(test_cloud_view._signer_is_valid(extra_dn))

# DNs corresponding to banned hosts should be invalid
# i.e not have POST rights
self.assertFalse(test_cloud_view._signer_is_valid(banned_dn))

# DNs corresponding to unknonw hosts should be invalid
# i.e not have POST rights
self.assertFalse(test_cloud_view._signer_is_valid(unknown_dn))

# mock the external call to the CMDB to retrieve the providers
# used in the _signer_is_valid method we are testing
# now we are mocking a failure of the CMDB to respond as expected
CloudRecordView._get_provider_list = Mock(return_value={})
# in which case we should reject all POST requests
self.assertFalse(test_cloud_view._signer_is_valid(allowed_dn))

0 comments on commit f1b2d10

Please sign in to comment.