Skip to content

FAQ

Jump to bottom Edit New page
David Fischer edited this page Dec 19, 2013 · 30 revisions

FAQ

Table of Contents

1   Disclaimer

This documentation is intended to be read by anyone with sufficient GNU/Linux / Ubuntu skills to understand what happens when executing the example of code snippets !

2   Documentation

2.1   How to generate David Fischer's Master Thesis ?

The Thesis is written in reStructuredText and generated by Sphinx and seamlessly handled by the script called generate-report.py which takes care of all the necessary to generate the report.

They are various targets that are already pre-programmed in the script and generate them according to your needs. Some of the parts of the report are extracted from the docstrings of the Python modules, this is why it is necessary to update the report according if you want to generate an up-to-date version of it. Of course it does involve a lot of work and it is probably not worth the effort to do so.

Warning

Please do not forget to install the project with the menu script first !

The help string:

cd OSCIED/scripts
python generate-doc.py --help
usage: generate-doc.py [-h] [--html] [--pdf] [--wiki]

optional arguments:
  -h, --help  show this help message and exit
  --html      Build HTML from source reStructuredText files (default: False)
  --pdf       Build PDF from source reStructuredText files (default: False)

Generate the report:

cd OSCIED/library
python generate-doc.py --html --pdf

Which outputs (maybe not up-to-date):

Install prerequisites
Fixes bitbucket.org/birkenfeld/sphinx/pull-request/98/fixes-typeerror-raised-from/diff
Fixed source file /usr/local/lib/python2.7/dist-packages/sphinx/writers/latex.py
Detect commit hash
Generate images from textual UMLs
Append hooks UMLs images together
Generate reStructuredText from templates
Generate links into common file
Append header files into common file
Build HTML from source reStructuredText files
Build PDF from source reStructuredText files

The build logs are stored into OSCIED/docs/david/master_thesis_rst/ named build_html.log and build_pdf.log. You will find a lot of warnings as we prefer to update the wiki and not the report according to the updates of the code.

If the process fails with the following error, please chmod the file or or run the script with sudo:

IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/python_dateutil-2.1-py2.7.egg/EGG-INFO/top_level.txt

3   Development

3.1   What the release number stands for ?

I do my best to follow this interesting recommendation : Semantic Versioning 2.0.0

3.2   How to run tests and check coverage ?

The core library of the project is developed in Python and a standard setup script is provided with it. Most of the base functions are documented and tested by nose through their docstrings. The most complex and less human readable tests of the code are written as separated scripts into the tests directory:

cd OSCIED/library
python setup.py test
xdg-open tests/cover/index.html

3.3   What/where are the main source code files ?

Here are listed the main entry-points (classes, files) that are useful for the developers working on this project.

3.3.1   Common parts of the Library

3.3.2   Deployment scenarios

3.3.3   IBC2013 deployment scenario

Implementation of a client script scenario.py using the various client-side classes of the library in order to deploy & manage OSCIED_. This script will read various configuration values from scenario_config.py.

Following files are read by the script to initialize the deployed plaftorms:

The scenario generates some statistics and charts embedded into a web-page called view.html

Note

You can use chartreux.py to generate nice charts about this demo: A real deployment is not required !

3.3.4   Orchestration service

3.3.5   Publication service

3.3.6   Storage service

3.3.7   Transformation service

3.3.8   Web user interface

3.4   How to add a new encoder to the platform ?

3.4.1   models.py

Add the name of the encoder into the tuple ENCODERS_NAMES.

Update the class TransformProfile for example add the necessary algorithm to select the output filename (method output_filename()) based on the input file name and the profile itself. This method can be used by your own client [5] to select the right output name when launching a transform task.

[5] See How OSCIED is used remotely, automatically driven by external code ?

3.4.2   config.yaml

Add options related to your encoder, this include adding an option to select the origin of your encoder (source repository, archive, local archive, ...).

3.4.3   TransformHooks.py

Update the class TransformHooks for example add the necessary steps into the setup method (hook_install()) to install your encoder from any source you want. The uninstall method (only called by the install hook itself) may be the right place to add cleanup code. Please make the install hook idempotent as described in the documentation of JuJu.

3.4.4   TransformWorker.py

This is the tricky part.

Add the necessary process management for your encoder into the method that implement the transformation task (obviously named transform_task). This include starting the encoder in a process and parsing the output to update the status of the task during his progression. You also should handle the way your encoder will return any exception and make them as understandable as possible by raising an exception with a message related to the error.

If you think that this method miss some parameters or your workflow is really different then you have two options:

  1. Update the model of the transform task.
  2. Create a new model & method for your new type of transformation task and implement new API methods into the orchestration service (api/, oscied-orchestra/).

3.5   How to add a new distribution point ?

There is a lot of room for improvement there.

Here are listed the main files you certainly have to update in order to enable the usage of a cloud-storage / CDN service:

3.5.1   models.py

Add a new model for your specific type of publication task or update the already available model.

3.5.2   config.yaml

Add any option that is worth for the configuration of the publication charm.

3.5.3   PublisherHooks.py

Update the class PublisherHooks for example add the necessary steps into the setup method (hook_install()) to install required libraries. The uninstall method (only called by the install hook itself) may be the right place to add cleanup code. Please make the install hook idempotent as described in the documentation of JuJu.

3.5.4   PublisherWorker.py

This is the tricky part.

Add the necessary code into the method that implement the publication task (obviously named publisher_task). This include the usage of the chosen cloud-storage / CDN service to make the media asset available to the end-users. You also should handle the way the service provied will return any exception and make them as understandable as possible by raising an exception with a message related to the error.

If you think that this method miss some parameters or your workflow is really different then you have two options:

  1. Update the model of the publication task.
  2. Create a new model & method for your new type of publication task and implement new API methods into the orchestration service (api/, oscied-orchestra/).

3.6   How OSCIED is used remotely, automatically driven by external code ?

TODO

https://github.com/ebu/OSCIED/tree/master/scenarios

4   Operational

4.1   How to deploy OSCIED ?

Please read the Deployment-guide.

4.2   In which way OSCIED can be deployed ?

There are 4 deployment scenarios for installing and using OSCIED SaaS in private and public clouds (more options not excluded), this platform can be deployed on top of [1]:

  1. A desktop computer for development purposes [2].
  2. A set of servers managed as a bare-metal cluster with MAAS or through an IaaS layer like OpenStack.
  3. The IaaS layer of a public cloud provider.
  4. Private (2) with public (3) in parallel [3].
  5. An hybrid setup consisting of the platform deployed in the private setup (2) and extended with encoding and distribution services in the public clouds (3).

You can integrate OSCIED in your own infrastructure on different levels:

  1. By connecting your own SAN instead of using OSCIED storage service
  2. Using the API to connect to your own MAM system

4.2.1   Pluggable Design

The charms of the project follows a standardized form of interface that makes possible the replacement of some services (for example the storage) by any compatible charm of the store. It also enable the possibility to deploy and connect the platform to other charms providing services like reverse proxies, ...

[1] See JuJu deploy to have a complete list of the targets handled by JuJu.
[2] The local provider where the charms are isolated as LXC containers.
[3] Shown during IBC 2013, the platforms are managed by a smart client via the API.

4.3   How to setup my cluster with MAAS ?

This answer is a short checklist of what to do in order to deploy a cluster managed with Canonical's MAAS. This guide, intentionally kept short, describes the deployment procedure at European Broadcasting Union (EBU/UER). Please read the official documentation for a more in-depth tutorial.

  1. Prepare your servers:
  • Install servers
  • Allocate servers in one network
  • Configure the BIOS of the servers to enable Wake-on Lan, Virtualization Extension, ...
  1. Install the MAAS controller as a guest of another computer [4]:
  • Use a separate laptop/desktop machine (any OS) that can access the servers from step 1 for:
  • Start VirtualBox and create a virtual machine (VM) to host the MAAS cluster controller
  • Configure the VM to boot from Ubuntu server Live CD image
  • Start the VM and choose in Ubuntu's splash menu to create a new MAAS controller
  • Carefully follow the installation guide from the official documentation until MAAS is configured
  1. Register your servers for the provisioning:
  • Boot your physical severs and accept them in MAAS Web user interface

Then your MAAS cluster is ready and can be registered as a target for JuJu to deploy your charms. Now you can follow the Deployment-guide to use your freshly installed cluster to deploy OSCIED in a bare-metal fashion!

[4] Not only to saves one computer but also to make it possible to snapshot the state of your controller.

4.4   How the scenarios switching is handled ?

The problem:

The JuJu command-line utility is programmed to read the required configuration files stored into your home directory. This make it difficult (tricky) to switch the context as every scenario have his own configuration.

The files:

~/.juju/environments.yaml
~/.juju/amazon.pem...
~/.ssh/id_rsa
~/.ssh/id_rsa.pub

The solution is a smart menu scripts that takes care of:

  • Enable the config. files from the scenario directory (if any) by copying them to the right directories.
  • Generating and enabling the missing config. files and copying them into the scenario directory.
  • Absolutely not overwriting your precious files (the ones in your home folder) by renaming them based on their SHA-1 sum. For example id_rsa may be renamed id_rsa_53ee9e4780da2d076e2bfd9a36a3e344 or id_rsa_72a8ffc740e223e777df9f4f37ec1555 depending of the file's content. By-design this solution ensure that you do not have a bunch of duplicate backups ... And yes, for instance the related public key id_rsa.pub will be renamed with the same prefix.
Add a custom footer
Add a custom sidebar
Clone this wiki locally