Merge branch 'cuds-3.0' into 'master'

Cuds 3.0

Closes #87, #30, #25, #19, #12, and #10

See merge request simphony/osp-core!11

.gitattributes

@@ -0,0 +1,3 @@
+* text=auto eol=lf
+*.{cmd,[cC][mM][dD]} text eol=crlf
+*.{bat,[bB][aA][tT]} text eol=crlf

.gitignore

@@ -2,9 +2,12 @@ __pycache__/
 build/
 cuds/__pycache__/
 cuds/classes/generated/
-cuds/metatools/__pycache__/
+cuds/generator/__pycache__/
 cuds/ontology/__pycache__/
 cuds/ontology/tools/__pycache__/
 simphony.egg-info/
 *~*
-
+.vscode/
+.mypy_cache/
+coverage.xml
+.coverage

.gitlab-ci.yml

@@ -2,11 +2,12 @@ image: "registry.gitlab.cc-asp.fraunhofer.de:4567/simphony/osp-core/osp-core-ci-
 
 stages:
   - build
+  #- test
 
 build_job:
   stage: build
   script:
-    - python3 setup.py install
-    - flake8 cuds/ --exclude=cuds/classes/generated,build --ignore=F401,F403,E402
+    - python3 setup.py install -o city
+    - flake8 cuds/ --exclude=cuds/classes/generated,*/__init__.py,build --ignore=W503
     - coverage run setup.py test
     - coverage report --omit=cuds/classes/generated/*,cuds/testing/* --skip-covered

Dockerfile

@@ -3,4 +3,4 @@ ADD . /code/libs
 WORKDIR /code/libs
 
 RUN pip install -r requirements.txt  --trusted-host pypi.org --trusted-host files.pythonhosted.org
-RUN python setup.py install
+RUN python setup.py install -c

README.md

@@ -1,95 +1,165 @@
+[![pipeline status](https://gitlab.cc-asp.fraunhofer.de/simphony/osp-core/badges/master/pipeline.svg)](https://gitlab.cc-asp.fraunhofer.de/simphony/osp-core/commits/master)
+[![coverage report](https://gitlab.cc-asp.fraunhofer.de/simphony/osp-core/badges/master/coverage.svg)](https://gitlab.cc-asp.fraunhofer.de/simphony/osp-core/commits/master)
 
-![pipeline_badge](https://gitlab.cc-asp.fraunhofer.de/simphony/osp-core/badges/master/pipeline.svg)
-![coverage_badge](https://gitlab.cc-asp.fraunhofer.de/simphony/osp-core/badges/master/coverage.svg)
+# OSP core
 
-# Core-OSP
-Native implementation of the core cuds object and the class generation by the SimPhoNy team at Fraunhofer IWM.
-Builds up on the previous version, simphony-common (SimPhoNy, EU FP7 Project (Nr. 604005) www.simphony-project.eu)
+Native implementation of the core cuds object and the class generation
+by the SimPhoNy team at Fraunhofer IWM. Builds up on the previous
+version, simphony-common (SimPhoNy, EU FP7 Project (Nr. 604005)
+www.simphony-project.eu)
 
-Copyright (c) 2018, Adham Hashibon and Materials Informatics Team at Fraunhofer IWM.
-All rights reserved.
-Redistribution and use are limited to the scope agreed with the end user.
-No parts of this software may be used outside of this context.
-No redistribution is allowed without explicit written permission.
+Copyright (c) 2018, Adham Hashibon and Materials Informatics Team at
+Fraunhofer IWM. All rights reserved. Redistribution and use are limited
+to the scope agreed with the end user. No parts of this software may be
+used outside of this context. No redistribution is allowed without
+explicit written permission.
 
 ## Requirements
-- PyYaml (on Windows, use https://stackoverflow.com/a/33673823)
 
-\*To run cuds2dot, `graphviz` must be installed in the system:
-```
-    sudo apt-get install graphviz
-```
-If you want to visualise the graph in a more interactive way than just the png, `xdot` can be used:
-```
-    sudo apt-get install xdot
-```
+- PyYaml (on Windows, use <https://stackoverflow.com/a/33673823>) for parsing yaml files
+- numpy for vector attributes of cuds
+- websockets for the transport layer
+- sqlalchemy for the sqlalchemy wrapper
+- requests for sending cuds to a server
+- unittest2 to run unittests
+- pympler for the performance test
+- responses for unittesting requests
+
 ## Installation
-The package requires python 3 (tested for 3.7), installation is based on setuptools:
+
+The package requires python 3 (tested for 3.6), installation is based on
+setuptools:
+
+```sh
+# build and install
+python3 setup.py install
 ```
-    # build and install
-    python3 setup.py install
 
-    # using toy ontology
-    python3 setup.py install -t
+```sh
+# using own ontology
+python3 setup.py install -o <path/to/ontology.own-ontology.yml>
 ```
+
 or:
+
+```sh
+# build for in-place development
+python3 setup.py develop
 ```
-    # build for in-place development
-    python3 setup.py develop
-```
+
+### Installation of OWL ontologies
+
+See doc/conversion_owl_to_yaml.md for working with an OWL ontology. \
+See doc/working_with_emmo.md for working with the EMMO.
 
 ## Testing
+
 Testing is included in setuptools:
-```
-    # run tests automatically
-    python3 setup.py test
+
+```sh
+# run tests automatically
+python3 setup.py test
 ```
 
 They can also be run manually:
-```
-    # manually run tests
-    python3 -m cuds.testing.test_api
+
+```sh
+# manually run tests
+python3 -m cuds.testing.test_api
 ```
 
-## Entry points
-There is an entry point for the *cuds2dot* tool for graph generation. Use `cuds2dot -h` for help.
+## Documentation
 
-The *class generator* can also be run with different ontologies and/or template files.
-The idea is that the users can define their own ontology and compare with the stable one.
-The new one will not be added to the python site-packages, and can be imported through relative paths.
+### API
+A standard, simple API has to be defined for the user to interact with OSP:
 
-If the generated classes are to be added to python for easier importing, the output folder can be set to `site-packages`:
 ```python
-    import site
-    print(site.getsitepackages())[0] + "/cuds/classes/generated")
+  import cuds.classes
+
+  a_cuds_object = cuds.classes.ACudsObject()
+  a_relationship = cuds.classes.ARelationship
+  a_cuba_key = cuds.classes.CUBA.A_CUBA_KEY
+
+  # These will also add the opposed relationship to the new contained entity
+  a_cuds_object.add(*other_cuds, rel=a_relationship)
+  a_cuds_object.add(yet_another_cuds)                          # Defaults to HAS_PART relationship
+
+  a_cuds_object.get()                                          # Returns the list of all the entities
+  a_cuds_object.get(rel=a_relationship)                        # Returns the list of the entities under that relationship
+  a_cuds_object.get(*uids)                                     # Searches through all the relationships for the uids
+  a_cuds_object.get(*uids, rel=a_relationship)                 # Faster, can filter through the relationship
+  a_cuds_object.get(cuba_key=a_cuba_key)                       # Returns the list of all the entities of that type
+  a_cuds_object.get(rel=a_relationship, cuba_key=a_cuba_key)   # Returns the list of all the entities of that type under the given relationship
+
+  # These will trigger the update in the opposed relationship of the erased element
+  a_cuds_object.remove()                                       # Removes all
+  a_cuds_object.remove(*uids/cuds_objects)                     # Searches through all the relationships for the uids/objects to remove
+  a_cuds_object.remove(*uids/cuds_objects, rel=a_relationship) # Faster, can filter through the relationship
+  a_cuds_object.remove(rel=a_relationship)                     # Delete all elements under a relationship
+  a_cuds_object.remove(cuba_key=a_cuba_key)                    # Delete all elements of a certain type
+  a_cuds_object.remove(rel=a_relationship, cuba_key=a_cuba_key)# Delete all elements of a certain type under the given relationship
+
+  a_cuds_object.update(*cuds_objects)                          # Searches through all the relationships for the objects to update
+
+  a_cuds_object.iter()                                         # Iterates through all
+  a_cuds_object.iter(cuba_key=a_cuba_key)                      # Iterates filtering by the object type
+  a_cuds_object.iter(rel=a_relationship)                       # Iterates filtering by the relationship
 ```
 
-## Directory structure
+### Data structure
+
+Each cuds object is represented as dictionary.
+This contains all the relationships of the cuds object.
+
+For example:
+
+```py
+{
+    CUBA.REL1: {"obj-uid-001": CUBA.OBJ1, "obj-uid-002": CUBA.OBJ2},
+    CUBA.REL2: {"obj-uid-003": CUBA.OBJ3, ...},
+    ...
+}
+```
+
+The related cuds object are referenced by their unique id, the uid.
+Each cuds object corresponds to a session.
+The session contains a registry that maps every uid to the corresponding cuds object.
+Each wrapper has a corresponding session. The default session is an instance of CoreSession.
+
+The attributes are fields of the cuds object:
+
+```py
+>>> c = cuds.classes.City(name="Freiburg")
+>>> c.name
+'Freiburg'
+```
+
+### Examples
+
+Further examples can be found in the /examples folder. There the usage of wrappers is explained.
+
+### Directory structure
+
 - cuds -- files necessary for the creation and usage of the cuds.
   - classes -- python classes required for using the cuds.
     - core -- common low level classes and utility code.
     - generated -- generated native cuds implementations.
-  - metatools -- class generator and template file.
+  - generator -- class generator and template file.
   - ontology -- ontological representation of the cuds.
-    - tools -- scripts that work on the ontology files
+  - session -- Different abstract classes for wrappers.
   - testing -- unittesting of the code.
 - doc -- documentation related files.
 - examples -- examples of usage.
 
-## Data structure
-Each cuds object is represented as a registry (dictionary).
-
-Each entry will enclose the elements contained by said object with the same CUBA key.
+### Architecture
 
-These will be again in a registry, this time by UID.
+The main components, their sub-components, and the interactions between them are given in the following diagrams.
 
-For example:
-```json
-    a_cuds_object = {
+The structure of a pure python OSP case:
+![Standalone CUDS](doc/standalone_cuds.svg)
 
-        "CUBA_key1": {"uid1": cuds_object1, "uid2": cuds_object2},
-        "CUBA_key2": {"uid3": cuds_object3},
-        "CUBA_key3": {"uid4": cuds_object4, "uid5": cuds_object5, "uid6": cuds_object6}
+The structure of a local wrapper:
+![Local CUDS](doc/local_cuds.svg)
 
-    }
-```
+The structure of a wrapper with a back-end in a remote server:
+![Distributed CUDS](doc/distributed_cuds.svg)

cuds/classes/__init__.py

@@ -1,2 +1,2 @@
-from .generated import *
-from .core.data_container import DataContainer
+from cuds.classes.generated import *
+from cuds.classes.cuds import Cuds