Skip to content
This repository has been archived by the owner. It is now read-only.
Permalink
Browse files
ARIA-22 Add CLI commands for parser
  • Loading branch information
tliron committed Nov 17, 2016
1 parent 3895f8c commit 33238199fcf6db59b11cc778d7f61609031bde8f
Show file tree
Hide file tree
Showing 11 changed files with 599 additions and 99 deletions.
188 README.md
@@ -1,4 +1,188 @@
Aria
ARIA
====

See http://ariatosca.org/
[ARIA](http://ariatosca.org/) is a minimal TOSCA orchestrator, as well as a platform for building
TOSCA-based products. Its features can be accessed via a well-documented Python API.

On its own, ARIA provides built-in tools for blueprint validation and for creating ready-to-run
service instances.

ARIA adheres strictly and meticulously to the
[TOSCA Simple Profile v1.0 cos01 specification](http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.0/cos01/TOSCA-Simple-Profile-YAML-v1.0-cos01.html),
providing state-of-the-art validation at seven different levels:

<ol start="0">
<li>Platform errors. E.g. network, hardware, or even an internal bug in ARIA (let us know,
please!).</li>
<li>Syntax and format errors. E.g. non-compliant YAML, XML, JSON.</li>
<li>Field validation. E.g. assigning a string where an integer is expected, using a list instead of
a dict.</li>
<li>Relationships between fields within a type. This is "grammar" as it applies to rules for
setting the values of fields in relation to each other.</li>
<li>Relationships between types. E.g. referring to an unknown type, causing a type inheritance
loop.</li>
<li>Topology. These errors happen if requirements and capabilities cannot be matched in order to
assemble a valid topology.</li>
<li>External dependencies. These errors happen if requirement/capability matching fails due to
external resources missing, e.g. the lack of a valid virtual machine, API credentials, etc.
</li>
</ol>

Validation errors include a plain English message and when relevant the exact location (file, row,
column) of the data the caused the error.

The ARIA API documentation always links to the relevant section of the specification, and likewise
we provide an annotated version of the specification that links back to the API documentation.


Quick Start
-----------

You need Python 2.6 or 2.7. Python 3+ is not currently supported.

To install, we recommend using [pip](https://pip.pypa.io/) and a
[virtualenv](https://virtualenv.pypa.io/en/stable/). In Debian-based systems:

sudo apt install python-setuptools
sudo -H easy_install pip
sudo -H pip install virtualenv

To install the latest development snapshot of ARIA:

virtualenv env
. env/bin/activate
pip install git+http://git-wip-us.apache.org/repos/asf/incubator-ariatosca.git

To test it, let's create a service instance from a TOSCA blueprint:

aria parse blueprints/tosca/node-cellar/node-cellar.yaml

You can also get it in JSON or YAML formats:

aria parse blueprints/tosca/node-cellar/node-cellar.yaml --json

Or get an overview of the relationship graph:

aria parse blueprints/tosca/node-cellar/node-cellar.yaml --graph

You can provide inputs as JSON, overriding default values provided in the blueprint

aria parse blueprints/tosca/node-cellar/node-cellar.yaml --inputs='{"openstack_credential": {"user": "username"}}'

Instead of providing them explicitly, you can also provide them in a file or URL, in either JSON or
YAML. If you do so, the value must end in ".json" or ".yaml":

aria parse blueprints/tosca/node-cellar/node-cellar.yaml --inputs=blueprints/tosca/node-cellar/inputs.yaml


CLI
---

Though ARIA is fully exposed as an API, it also comes with a CLI tool to allow you to work from the
shell:

aria parse blueprints/tosca/node-cellar/node-cellar.yaml instance

The `parse` command supports the following directives to create variations of the default consumer
chain:

* `presentation`: emits a colorized textual representation of the Python presentation classes
wrapping the blueprint.
* `model`: emits a colorized textual representation of the complete service model derived from the
validated blueprint. This includes all the node templates, with their requirements satisfied at
the level of relating to other node templates.
* `types`: emits a colorized textual representation of the type hierarchies.
* `instance`: **this is the default command**; emits a colorized textual representation of a
service instance instantiated from the service model. Here the node templates are each used to
create one or more nodes, with the appropriate relationships between them. Note that every time
you run this consumer, you will get a different set of node IDs. Use `--graph` to see just the
node relationship graph.

For all these commands, you can also use `--json` or `--yaml` flags to emit in those formats.

Additionally, The CLI tool lets you specify the complete classname of your own custom consumer to
chain at the end of the default consumer chain, after `instance`.

Your custom consumer can be an entry point into a powerful TOSCA-based tool or application, such as
an orchestrator, a graphical modeling tool, etc.


Development
-----------

Instead of installing with `pip`, it would be easier to work directly with the source files:

pip install virtualenv
virtualenv env
. env/bin/activate
git clone http://git-wip-us.apache.org/repos/asf/incubator-ariatosca.git ariatosca
cd ariatosca
pip install -e .

To run tests:

pip install tox
tox

Here's a quick example of using the API to parse YAML text into a service instance:

from aria import install_aria_extensions
from aria.parser.consumption import ConsumptionContext, ConsumerChain, Read, Validate, Model, Instance
from aria.parser.loading import LiteralLocation

def parse_text(payload, file_search_paths=[]):
context = ConsumptionContext()
context.presentation.location = LiteralLocation(payload)
context.loading.file_search_paths += file_search_paths
ConsumerChain(context, (Read, Validate, Model, Instance)).consume()
if not context.validation.dump_issues():
return context.modeling.instance
return None

install_aria_extensions()

print parse_text("""
tosca_definitions_version: tosca_simple_yaml_1_0
topology_template:
node_templates:
MyNode:
type: tosca.nodes.Compute
""")


Parser API Architecture
-----------------------

ARIA's parsing engine comprises individual "consumers" (in the `aria.parser.consumption` package)
that do things with blueprints. When chained together, each performs a different task, adds its own
validations, and can provide its own output.

Parsing happens in five phases, represented in five packages:

* `aria.parser.loading`: Loaders are used to read the TOSCA data, usually as text. For example
UriTextLoader will load text from URIs (including files).
* `aria.parser.reading`: Readers convert data from the loaders into agnostic raw data. For
example, `YamlReader` converts YAML text into Python dicts, lists, and primitives.
* `aria.parser.presentation`: Presenters wrap the agnostic raw data in a nice
Python facade (a "presentation") that makes it much easier to work with the data, including
utilities for validation, querying, etc. Note that presenters are _wrappers_: the agnostic raw
data is always maintained intact, and can always be accessed directly or written back to files.
* `aria.parser.modeling.model`: Here the topology is normalized into a coherent structure of
node templates, requirements, and capabilities. Types are inherited and properties are assigned.
The service model is a _new_ structure, which is not mapped to the YAML. In fact, it is possible
to generate the model programmatically, or from a DSL parser other than TOSCA.
* `aria.parser.modeling.instance`: The service instance is an instantiated service model. Node
templates turn into node instances (with unique IDs), and requirements are satisfied by matching
them to capabilities. This is where level 5 validation errors are detected (see above).

The phases do not have to be used in order. Indeed, consumers do not have to be used at all: ARIA
can be used to _produce_ blueprints. For example, it is possible to fill in the
`aria.parser.presentation` classes programmatically, in Python, and then write the presentation
to a YAML file as compliant TOSCA. The same technique can be used to convert from one DSL (consume
it) to another (write it).

The term "agnostic raw data" (ARD?) appears often in the documentation. It denotes data structures
comprising _only_ Python dicts, lists, and primitives, such that they can always be converted to and
from language-agnostic formats such as YAML, JSON, and XML. A considerable effort has been made to
conserve the agnostic raw data at all times. Thus, though ARIA makes good use of the dynamic power
of Python, you will _always_ be able to use ARIA with other systems.
@@ -17,6 +17,9 @@
Aria top level package
"""

import sys
import pkgutil

from .VERSION import version as __version__

from .orchestrator.decorators import workflow, operation
@@ -38,6 +41,23 @@
_resource_storage = {}


def install_aria_extensions():
"""
Iterates all Python packages with names beginning with :code:`aria_extension_` and calls
their :code:`install_aria_extension` function if they have it.
"""

for loader, module_name, _ in pkgutil.iter_modules():
if module_name.startswith('aria_extension_'):
module = loader.find_module(module_name).load_module(module_name)

if hasattr(module, 'install_aria_extension'):
module.install_aria_extension()

# Loading the module has contaminated sys.modules, so we'll clean it up
del sys.modules[module_name]


def application_model_storage(driver):
"""
Initiate model storage for the supplied storage driver
@@ -20,6 +20,8 @@
import argparse
from functools import partial

from ..utils.argparse import ArgumentParser

NO_VERBOSE = 0


@@ -48,10 +50,6 @@ def _wrapper(parser):
action='count',
default=NO_VERBOSE,
help='Set verbosity level (can be passed multiple times)')
sub_parser.add_argument(
'-d', '--deployment-id',
required=True,
help='A unique ID for the deployment')
func(sub_parser)
return sub_parser
return _wrapper
@@ -61,14 +59,16 @@ def config_parser(parser=None):
"""
Top level argparse configuration
"""
parser = parser or argparse.ArgumentParser(
prog='Aria',
description="Aria's Command Line Interface",
parser = parser or ArgumentParser(
prog='ARIA',
description="ARIA's Command Line Interface",
formatter_class=SmartFormatter)
parser.add_argument('-v', '--version', action='version')
sub_parser = parser.add_subparsers(title='Commands', dest='command')
add_init_parser(sub_parser)
add_execute_parser(sub_parser)
add_parse_parser(sub_parser)
add_spec_parser(sub_parser)
return parser


@@ -80,6 +80,10 @@ def add_init_parser(init):
"""
``init`` command parser configuration
"""
init.add_argument(
'-d', '--deployment-id',
required=True,
help='A unique ID for the deployment')
init.add_argument(
'-p', '--blueprint-path',
dest='blueprint_path',
@@ -109,6 +113,10 @@ def add_execute_parser(execute):
"""
``execute`` command parser configuration
"""
execute.add_argument(
'-d', '--deployment-id',
required=True,
help='A unique ID for the deployment')
execute.add_argument(
'-w', '--workflow',
dest='workflow_id',
@@ -132,3 +140,62 @@ def add_execute_parser(execute):
default=1,
type=int,
help='How many seconds to wait before each task is retried')


@sub_parser_decorator(
name='parse',
help='Parse a blueprint',
formatter_class=SmartFormatter)
def add_parse_parser(parse):
"""
``parse`` command parser configuration
"""
parse.add_argument(
'uri',
help='URI or file path to profile')
parse.add_argument(
'consumer',
nargs='?',
default='instance',
help='consumer class name (full class path or short name)')
parse.add_argument(
'--loader-source',
default='aria.parser.loading.DefaultLoaderSource',
help='loader source class for the parser')
parse.add_argument(
'--reader-source',
default='aria.parser.reading.DefaultReaderSource',
help='reader source class for the parser')
parse.add_argument(
'--presenter-source',
default='aria.parser.presentation.DefaultPresenterSource',
help='presenter source class for the parser')
parse.add_argument(
'--presenter',
help='force use of this presenter class in parser')
parse.add_argument(
'--prefix', nargs='*',
help='prefixes for imports')
parse.add_flag_argument(
'debug',
help_true='print debug info',
help_false='don\'t print debug info')
parse.add_flag_argument(
'cached-methods',
help_true='enable cached methods',
help_false='disable cached methods',
default=True)


@sub_parser_decorator(
name='spec',
help='Specification tool',
formatter_class=SmartFormatter)
def add_spec_parser(spec):
"""
``spec`` command parser configuration
"""
spec.add_argument(
'--csv',
action='store_true',
help='output as CSV')

0 comments on commit 3323819

Please sign in to comment.