COVESA · SebastianSchildt · Jul 13, 2022 · Jun 7, 2022 · Jun 11, 2022 · Jun 11, 2022
diff --git a/README.md b/README.md
@@ -16,13 +16,13 @@ Examples on tool usage can be found in the [VSS Makefile](https://github.com/COV
 
  Tool | Description | Type of Tool | Documentation |
 | ------------------ | ----------- | -------------------- |-------------------- |
-| [vspec2x.py](vspec2x.py) | Parses and expands VSS into different text based output formats. Currently supports `json`, `yaml`,`csv`,`idl`  | Community Supported | `./vspec2x --help` |
-[vspec2csv.py](vspec2csv.py) | Shortcut for [vspec2x.py](vspec2x.py) generating CSV output | Community Supported | -|
-[vspec2ddsidl.py](vspec2ddsidl.py) | Shortcut for [vspec2x.py](vspec2x.py) generating DDS-IDL output | Community Supported | [Documentation](docs/VSS2DDSIDL.md) |
-[vspec2json.py](vspec2json.py) |  Shortcut for [vspec2x.py](vspec2x.py) generating JSON output | Community Supported | - |
-[vspec2yaml.py](vspec2yaml.py) | Shortcut for [vspec2x.py](vspec2x.py) generating flattened YAML output | Community Supported | - |
-[vspec2binary.py](vspec2binary.py) | The binary toolset consists of a tool that translates the VSS YAML specification to the binary file format (see below), and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go | Community Supported | [Documentation](binary/README.md) | 
-[vspec2franca.py](vspec2franca.py) | Parses and expands a VSS and generates a Franca IDL specification | Community Supported | - |
+| [vspec2x.py](vspec2x.py) | Parses and expands VSS into different text based output formats. Currently supports `json`, `yaml`,`csv`,`idl`  | Community Supported | Try `./vspec2x --help` or check [vspec2x documentation](docs/vspec2x.md) |
+[vspec2csv.py](vspec2csv.py) | Shortcut for [vspec2x.py](vspec2x.py) generating CSV output | Community Supported |  Check [vspec2x documentation](docs/vspec2x.md) |
+[vspec2ddsidl.py](vspec2ddsidl.py) | Shortcut for [vspec2x.py](vspec2x.py) generating DDS-IDL output | Community Supported | [VSS2DDSIDL Documentation](docs/VSS2DDSIDL.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) |
+[vspec2json.py](vspec2json.py) |  Shortcut for [vspec2x.py](vspec2x.py) generating JSON output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
+[vspec2yaml.py](vspec2yaml.py) | Shortcut for [vspec2x.py](vspec2x.py) generating flattened YAML output | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
+[vspec2binary.py](vspec2binary.py) | The binary toolset consists of a tool that translates the VSS YAML specification to the binary file format (see below), and two libraries that provides methods that are likely to be needed by a server that manages the VSS tree, one written in C, and one in Go | Community Supported | [vspec2binary Documentation](binary/README.md). For general parameters check [vspec2x documentation](docs/vspec2x.md) | 
+[vspec2franca.py](vspec2franca.py) | Parses and expands a VSS and generates a Franca IDL specification | Community Supported | Check [vspec2x documentation](docs/vspec2x.md) |
 [test_contants.py](tests/model/test_contants.py) | Tool used for internal testing  | Community Supported | - |
 [test_vsstree.py](tests/model/test_vsstree.py) | Tool used for internal testing | Community Supported | - |
 [vspec2c.py](contrib/vspec2c.py) | The vspec2c tooling allows a vehicle signal specification to be translated from its source YAML file to native C code that has the entire specification encoded in it. | Contrib (Obsolete, no longer functional) | [Documentation](contrib/vspec2c/README.md) |

diff --git a/contrib/vspec2protobuf.py b/contrib/vspec2protobuf.py
@@ -41,7 +41,7 @@ def print_message_body(nodes, proto_file):
     for i, node in enumerate(nodes, 1):
         data_type = node.qualified_name("")
         if not node.is_branch():
-            dt_val = node.data_type.value
+            dt_val = node.datatype.value
             data_type = mapped.get(dt_val.strip("[]"), dt_val.strip("[]"))
             data_type = ("repeated " if dt_val.endswith("[]") else "") + data_type
         proto_file.write(f"  {data_type} {node.name} = {i};" + "\n")

diff --git a/docs/vspec2x.md b/docs/vspec2x.md
@@ -0,0 +1,156 @@
+# vspec2x converters
+
+vspec2x is a family of VSS converters that share a common codebase.
+
+As a consequence it provides general commndline parameters guidng the parsing of vspec, as well as paramters specific to specific output formats.
+
+You can get a description of supported commandlines parameters by running `vspec2x.py --help`.
+
+This documentation will give some examples and elaborate more on specific parameters.
+
+The supported arguments might look like this
+ ```
+usage: vspec2x.py [-h] [-I dir] [-e EXTENDED_ATTRIBUTES] [-s] [--abort-on-unknown-attribute] [--abort-on-name-style]
+                  [--format format] [--no-uuid] [-o overlays] [--json-all-extended-attributes] [--json-pretty]
+                  [--yaml-all-extended-attributes] [-v version] [--all-idl-features] [--gqlfield GQLFIELD GQLFIELD]
+                  <vspec_file> <output_file>
+```
+
+A common commandline to convert the VSS standard catalogue into a JSON file is
+
+```
+% python vspec2x.py --format json  -I ../spec ../spec/VehicleSignalSpecification.vspec vss.js
+on
+Output to json format
+Known extended attributes: 
+Loading vspec from ../spec/VehicleSignalSpecification.vspec...
+Calling exporter...
+Generating JSON output...
+Serializing compact JSON...
+All done.
+```
+
+This assumes you checked out the [COVESA Vehicle Singal Specification](https://github.com/covesa/vehicle_signal_specification) which contains vss-tools including vspec2x as a submodule.
+
+The `-I` parameter adds a directory to search for includes referenced in you `.vspec` files. `-I` can be used multiple times to specify more include directories.
+
+The first positional argument - `../spec/VehicleSignalSpecification.vspec` in the example  - gives the (root) `.vspec` file to be converted. The second positional argument  - `vss.json` in the example - is the output file.
+
+The `--format` parameter determines the output format, `JSON` in our example. If format is omitted `vspec2x` tries to guess the correct output format based on the extension of the second positional argument. Alternatively vss-tools supports *shortcuts* for community supported exporters, e.g. `vspec2json.py` for generating JSON. The shortcuts really only add the `--format` parameter for you, so 
+
+```
+python vspec2json.py  json  -I ../spec ../spec/VehicleSignalSpecification.vspec vss.js
+```
+
+is equivalent to the example above.
+
+## General parameters
+
+### --abort-on-unknown-attribute
+Terminates parsing when an unknown attribute is encountered, that is an attribute that is not defined in the [VSS standard catalogue](https://covesa.github.io/vehicle_signal_specification/rule_set/), and not whitelisted using the extended attribute parameter `-e` (see below).
+
+*Note*: Here an *attribute* refers to VSS signal metadata sich as "datatype", "min", "max", ... and not to the VSS signal type attribute
+
+###  --abort-on-name-style
+Terminates parsing, when the name of a signal does not follow [VSS recomendations](https://covesa.github.io/vehicle_signal_specification/rule_set/basics/#naming-conventions).
+
+### --strict
+Equivalent to setting `--abort-on-unknown-attribute` and `--abort-on-name-style` 
+
+### --no-uuid
+Request the exporter to not output uuids. This setting may not apply to all exporters (e.g. there might be output that never generate uuids), but those that do, should honor the parameter.
+
+## Handling of overlays and extensions
+`vspec2x` allows composition of several overlays on top of a base vspec, to extend the model or overwrite certain metadata. Check [VSS documentation](https://covesa.github.io/vehicle_signal_specification/introduction/) on the concept of overlays. 
+
+To add overlays add one or more `-o` or  `--overlays` parameters, e.g.
+
+```
+python vspec2yaml.py -I ../spec ../spec/VehicleSignalSpecification.vspec  -o overlay.vspec  withoverlay.yml
+```
+
+You can also use VSS specifications with custom metadata not used in the standard catalogue, for example if your VSS model includes a `source` or `quality` metadata for sensors.
+
+As an example consider the following `overlay.vspec`:
+
+```yaml
+#
+# Example overlay
+#
+Vehicle:
+  type: branch
+
+Vehicle.Speed:
+    quality: 100
+    source: "ecu0xAA"
+```
+
+Will give a warning about unknown attributes, or even terminate the parsing when `-s`, `--strict`  or `--abort-on-unknown-attribute` is used
+
+```
+% python vspec2json.py -I ../spec ../spec/VehicleSignalSpecification.vspec --strict -o overlay.vspec   test.json
+Output to json format
+Known extended attributes: 
+Loading vspec from ../spec/VehicleSignalSpecification.vspec...
+Applying VSS overlay from overlay.vspec...
+Warning: Attribute(s) quality, source, lol in element Speed not a core or known extended attribute.
+You asked for strict checking. Terminating.
+```
+
+You can whitelist extended metadata attributes using the `-e` parameter with a comma sperated list of attributes
+
+```
+python vspec2json.py -I ../spec ../spec/VehicleSignalSpecification.vspec -e quality,source -o overlay.vspec   test.json
+```
+
+In this case the expectation is, that the general expectation is, that the generated output will contain the whitelisted extended metadata attributes, if the exporter supports them.
+
+__Note: Not all exporters (need to) support (all) extended metadata attributes!__ Currently, the `yaml` and `json` exporters support arbitrary metadata.
+
+## JSON exporter notes
+
+### --json-all-extended-attributes
+Lets the exporter generate _all_ extended metadata attributes found in the model. By default the exporter is generating only those given by the `-e`/`--extended-attributes` parameter.
+
+### --json-pretty
+If the paramter is set it will pretty-print the JSON output, otherwise you will get a minimized version
+
+## YAML exporter notes
+
+### --yaml-all-extended-attributes
+Lets the exporter generate _all_ extended metadata attributes found in the model. By default the exporter is generating only those given by the `-e`/`--extended-attributes` parameter.
+
+## CSV exporter notes
+The CSV exporter does not currently support the no-uuid option.
+
+## FRANCA exporter notes
+The Franca exporter does not currently support the no-uuid option.
+
+## DDS-IDL exporter notes
+
+### --all-idl-features
+Will also generate non-payload const attributes such as unit/datatype. Default is not to generate them/comment them out because at least Cyclone DDS and FastDDS do not support const. For more information check the [DDS-IDL exporter docs](VSS2DDSIDL.md).
+
+## GRAPHQL exporter notes
+
+### --gqlfield GQLFIELD GQLFIELD
+Add additional fields to the nodes in the graphql schema. use: <field_name> <description>
+
+
+## Writing your own exporter
+This is easy. Put the code in file in the [vssexporters directory](../vssexporters/). 
+
+Mandatory functions to be implemented are
+```python
+def add_arguments(parser: argparse.ArgumentParser):
+```
+
+and
+
+```python
+def export(config: argparse.Namespace, root: VSSNode):
+```
+See one of the existing exporters for an example.
+
+Add your exporter module to the `Exporter` class in [vspec2x.py](../vspec2x.py). 
+
diff --git a/tests/model/test_vsstree.py b/tests/model/test_vsstree.py
@@ -16,7 +16,7 @@ def test_simple_construction(self):
         self.assertEqual(VSSType.BRANCH, node.type)
         self.assertEqual("26d6e362-a422-11ea-bb37-0242ac130002", node.uuid)
         self.assertFalse(node.has_unit())
-        self.assertFalse(node.has_data_type())
+        self.assertFalse(node.has_datatype())
         self.assertFalse(node.is_private())
 
     def test_complex_construction(self):
@@ -31,15 +31,15 @@ def test_complex_construction(self):
         self.assertEqual("some desc", node.description)
         self.assertEqual(VSSType.SENSOR, node.type)
         self.assertEqual("26d6e362-a422-11ea-bb37-0242ac130002", node.uuid)
-        self.assertEqual(VSSDataType.UINT8, node.data_type)
+        self.assertEqual(VSSDataType.UINT8, node.datatype)
         self.assertEqual(Unit.KILOMETER, node.unit)
         self.assertEqual(0, node.min)
         self.assertEqual(100, node.max)
         self.assertEqual(["one", "two"], node.allowed)
         self.assertEqual(False, node.aggregate)
-        self.assertEqual("test-default", node.default_value)
+        self.assertEqual("test-default", node.default)
         self.assertTrue(node.has_unit())
-        self.assertTrue(node.has_data_type())
+        self.assertTrue(node.has_datatype())
         self.assertFalse(node.is_private())
 
     def test_private_attribute(self):
@@ -69,13 +69,13 @@ def test_merge_nodes(self):
         node_target = VSSNode("MyNode", target)
         node_source = VSSNode("Private", source)
         self.assertEqual("e36a1d8c-4d06-4c22-ba69-e8b39434a7a3", node_target.uuid)
-        self.assertTrue(node_target.has_data_type())
+        self.assertTrue(node_target.has_datatype())
         self.assertEqual("2cc90035-e1c2-43bf-a394-1a439addc8ad", node_source.uuid)
 
         node_target.merge(node_source)
         self.assertEqual("2cc90035-e1c2-43bf-a394-1a439addc8ad", node_target.uuid)
-        self.assertTrue(node_target.has_data_type())
-        self.assertEqual(VSSDataType.UINT8, node_target.data_type)
+        self.assertTrue(node_target.has_datatype())
+        self.assertEqual(VSSDataType.UINT8, node_target.datatype)
         self.assertEqual(Unit.KILOMETER, node_target.unit)
         self.assertEqual(0, node_target.min)
         self.assertEqual(100, node_target.max)

diff --git a/vspec/__init__.py b/vspec/__init__.py
@@ -25,7 +25,7 @@
 
 import deprecation
 
-from .model.vsstree import VSSNode
+from .model.vsstree import ImpossibleMergeException, IncompleteElementException, VSSNode
 
 
 class VSpecError(Exception):
@@ -84,13 +84,13 @@ def convert_yaml_to_list(raw_yaml):
     return lst
 
 
-def load_tree(file_name, include_paths, merge_private=False, break_on_noncore_attribute=False, break_on_name_style_violation=False, expand_inst=True):
+def load_tree(file_name, include_paths, merge_private=False, break_on_unknown_attribute=False, break_on_name_style_violation=False, expand_inst=True):
     flat_model = load_flat_model(file_name, "", include_paths)
     absolute_path_flat_model = create_absolute_paths(flat_model)
     deep_model = create_nested_model(absolute_path_flat_model, file_name)
     cleanup_deep_model(deep_model)
     dict_tree = deep_model["children"]
-    tree = render_tree(dict_tree, merge_private, break_on_noncore_attribute=break_on_noncore_attribute,
+    tree = render_tree(dict_tree, merge_private, break_on_unknown_attribute=break_on_unknown_attribute,
                        break_on_name_style_violation=break_on_name_style_violation)
     if expand_inst:
         expand_tree_instances(tree)
@@ -588,18 +588,18 @@ def yamilify_includes(text, is_list):
     return text
 
 
-def render_tree(tree_dict, merge_private=False, break_on_noncore_attribute=False, break_on_name_style_violation=False) -> VSSNode:
+def render_tree(tree_dict, merge_private=False, break_on_unknown_attribute=False, break_on_name_style_violation=False) -> VSSNode:
     if len(tree_dict) != 1:
         raise Exception('Invalid VSS model, must have single root node')
 
     root_element_name = next(iter(tree_dict.keys()))
     root_element = tree_dict[root_element_name]
-    tree_root = VSSNode(root_element_name, root_element, break_on_noncore_attribute=break_on_noncore_attribute,
+    tree_root = VSSNode(root_element_name, root_element, break_on_unknown_attribute=break_on_unknown_attribute,
                         break_on_name_style_violation=break_on_name_style_violation)
 
     if "children" in root_element.keys():
         child_nodes = root_element["children"]
-        render_subtree(child_nodes, tree_root, break_on_noncore_attribute=break_on_noncore_attribute,
+        render_subtree(child_nodes, tree_root, break_on_unknown_attribute=break_on_unknown_attribute,
                        break_on_name_style_violation=break_on_name_style_violation)
 
     if merge_private:
@@ -609,15 +609,20 @@ def render_tree(tree_dict, merge_private=False, break_on_noncore_attribute=False
     return tree_root
 
 
-def render_subtree(subtree, parent, break_on_noncore_attribute=False, break_on_name_style_violation=False):
+def render_subtree(subtree, parent, break_on_unknown_attribute=False, break_on_name_style_violation=False):
     for element_name in subtree:
         current_element = subtree[element_name]
 
-        new_element = VSSNode(element_name, current_element, parent=parent, break_on_noncore_attribute=break_on_noncore_attribute,
+        try:
+            new_element = VSSNode(element_name, current_element, parent=parent, break_on_unknown_attribute=break_on_unknown_attribute,
                               break_on_name_style_violation=break_on_name_style_violation)
+        except IncompleteElementException as e:
+            print(f"Invalid VSS: {e}")
+            print("Terminating.")
+            sys.exit(-1)
         if "children" in current_element.keys():
             child_nodes = current_element["children"]
-            render_subtree(child_nodes, new_element, break_on_noncore_attribute,
+            render_subtree(child_nodes, new_element, break_on_unknown_attribute,
                            break_on_name_style_violation=break_on_name_style_violation)
 
 
@@ -675,7 +680,11 @@ def merge_tree(base: VSSNode, overlay: VSSNode):
             # so children in base will not be overwritten
             #print(f"Merging {overlay_element.qualified_name()}")
             other_node: VSSNode = r.get(base, element_name)
-            other_node.merge(overlay_element)
+            try:
+                other_node.merge(overlay_element)
+            except ImpossibleMergeException as e:
+                print(f"Merging impossible: {e}")
+                sys.exit(-1)
 
 
 def create_tree_uuids(root: VSSNode):