Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Long-term draft topic: Vsc lib with fixes #31

Closed
wants to merge 41 commits into from
Closed

Long-term draft topic: Vsc lib with fixes #31

wants to merge 41 commits into from

Conversation

gunnarx
Copy link
Collaborator

@gunnarx gunnarx commented Aug 15, 2022

Fixed according to feedback on PR #30. I didn't force push that branch without @magnusfeuer's permission. In any case, I recommend we also squash these small fixes into the original commit, and that we should merge the PR (it still goes to separate branch). What do you think?

@gunnarx gunnarx force-pushed the vsc_lib_with_fixes branch 2 times, most recently from ab2150d to a559637 Compare August 15, 2022 15:20
erikbosch
erikbosch reviewed Aug 15, 2022
View reviewed changes
vsclib/LICENSE
@@ -0,0 +1,373 @@
Mozilla Public License Version 2.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same license as in root repo, right? So why do we need this file.

This was referenced Aug 19, 2022
Merged
Merged
@gunnarx gunnarx mentioned this pull request Sep 15, 2022
Closed
erikbosch
erikbosch reviewed Sep 16, 2022
View reviewed changes
vsclib/setup.py Outdated
description="VSC Generator Library",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/COVESA/vsclib",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this URL ever exist? Or should it be vsc-tools?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as of now it seems like it should be vsc-tools.

erikbosch
erikbosch reviewed Sep 16, 2022
View reviewed changes
vsclib/setup.py Outdated
classifiers=[
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"License :: Other/Proprietary License",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we use MPL2 - is the license statement here correct? Compare with e.g. https://github.com/mozilla/libmozdata/blob/master/setup.py

Also, here we explictly state Python 3.7 and 3.8, but in buildcheck we use Python 3.10. Is it needed to specify Python version here. (In the long run, but not now, we should preferably perform tests with all Python versions we claim to support. I.e. if we say here that Python 3.7 is sufficient then we shall have a build that verifies that 3.7 actually is sufficient)

Copy link
Collaborator Author

@gunnarx gunnarx Sep 25, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I agree, the license info looks wrong to me.

  2. Yeah the python version in the buildcheck has evolved over time. I seem to recall that it started with a language feature we used that required 3.9, and then the Ubuntu version of the GitHub CI environment underwent some change so it did not support the version we demanded, so it was then changed to 3.10 for the CI to run OK.
    In any case, good catch - we should make this consistent and also avoid overspecifying such details.

miketsukerman and others added 22 commits September 21, 2022 15:56
@miketsukerman @gunnarx
vsc-tools: refactor python module structure
3b004df 
* Add python setuptools setup.py script.
* Move python code to the vsc folder.
* Move templates folder to the vsc folder.
* Install vscgen tool to simplify codegenerator use via cmd.
* Update github actions (use python 3.10 and vscgen).

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vsc_ast: use python dataclasses for AST
deb95e9 
* Implement vehicle service catalog AST using python dataclasses.
* Implement parsing of YAML vsc format into vsc AST format.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
get_tests: update vsc-tools python tests
a918d9d 
* Add simple tests for newly written dataclasses AST.
* Add new example YAML files input2.yaml and input3.yaml.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
.gitignore: extend list by python default temporary files
cb58613 
Take over typic gitignore for python projects from
https://github.com/github/gitignore/blob/main/Python.gitignore

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@gunnarx
vsc_ast: Rewrite main parsing function
6fcff3e
@gunnarx
vsc_ast: Complete the AST classes
b47b9eb
@gunnarx
vsc_parser: Delegate to new vsc_ast module
0eadd84
@gunnarx
vsc_parser: Remove all unused functions
bc1cd40 
All the previous code in this file, to read YAML into an AST tree, is
now superseded by vsc_ast.py.

There are a few files and scripts referring to this file so at the
moment it remains as a wrapper around vst_ast but soon it should
probably be replaced completely.
@gunnarx
vsc_generator: Update to match new vst_ast parser
0ac8bef
@gunnarx
vsc_generator: Formatting and cleaning
a000667 
(should be no functional change)
@gunnarx
tests: Update for current language model
916627b 
- Methods need to be inside a namespace
- includes: should be a list
- Change one test to output name of main file instead of first namespace

See also previous comment
@gunnarx
templates: Update to match current language model
b329dd5
@gunnarx
vsc_ast: Remove post_init initialization of data objects
ea32702 
From what I can tell, the new parsing function instantiates all needed
fields of all objects in the AST tree (or leaves the field as
the default (= None), if something is Optional and not provided by
the input YAML.

I therefore removed all the post_init functions, here as a separate
commit in case we need to test further, rollback, or something.

unit-tests and basic template generation seem to run fine (except for
those templates that need update, but that is known)
@gunnarx
GitHub Actions Temporary branch change
a49ec43
@miketsukerman @gunnarx
vscgen: update command line interface
d45cfd6 
In vscgen tool template argument becomes optional. When vscgen
<yaml.file> called it would use default templates from
vsc.templates.default_templates dictionary.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vsc_ast: bring back AST root node
65b527c 
After short discussion we have decided to bring back the AST node.
This node is always a root, but inherits everything from the Namespace class.

And vsc_parser.gen() function now accepts object of any type. Later we
might want to introduce some checks for the passed node, like
assert dataclass.is_dataclass(node).

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vsc_ast: move parsing code away from vsc_ast to vsc_parser
4f53efa 
Moving all code related to YAML processing to the vsc_parser.py
from vsc_ast.py to keep in vsc_ast module only AST nodes.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vsc_generator: make algorithms more abstract
f5cfeb2 
Use Any instead of Generic[T] for gen() function.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
setup.py: package template files in the vsc/templates folder
46cb12b 
Improve setup.py script to properly package and install files inside the
templates folder. Now setup.py will install the *.tpl and *.html files
into the respective python3 packages directory. This allows to use
module independently.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vsc_generator: remove code related to template engine (jinja2)
ce29f9a 
An attempt to reverse dependencies by moving all the jinja2 related code
to the vsc.templates module. This makes vsc_generator.py more abstract
and in theory removes dependency on the concrete template engine.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vsc_ast: cleanup module
7e03a24 
Making description field empty by default instead of None.
Removing Optional[..] from the Lists and using default_factory
assigning empty lists by default to avoid additional checks in the
templates code.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vsc_ast: return Optional[...]
442c696 
Bring back Optional[List[...]] and Optional[str] to some
of the fields to bring AST closer to the specification. It would
not change the runtime behaviour, lists would have default value [], etc.
Fix arraysize type in some of the classes from Optional[str]
to Optional[int].

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
.github: update github actions configuration
47cc325 
Change python version from 3.10.6 to 3.10.7.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
vcs_ast: add comments to the AST classes
b176b03 
Comments added to the AST classes, using information from
the VSC specification and format of python docstrings.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
README.md: cleanup
3fdd7e2 
I noticed some discrepancy in the readme file and rawly fixed all the
incosistencies I have been able to find out. Maybe there are bigger
changes needed, but I will leave it for the future PRs.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@miketsukerman @gunnarx
copyright: fix license headers
38d28a9 
Remove line "#All right reserved" from the files added by previous MR.

Signed-off-by: Mikhail Tsukerman <mikhail.tcukerman@daimler.com>
Signed-off-by: Magnus Feuer <magnus.feuer@mercedes-benz.com>
@gunnarx gunnarx self-assigned this Sep 25, 2022
@gunnarx gunnarx marked this pull request as draft September 26, 2022 14:59
erikbosch and others added 3 commits September 26, 2022 19:55
@erikbosch @gunnarx
Fix documentation regression
7a70f4c 
Signed-off-by: Erik Jaegervall <erik.jaegervall@se.bosch.com>
@gunnarx
README: Update all setup/installation instructions
6e1b25b
@erikbosch @gunnarx
Fix template regressions
7e30d68 
Signed-off-by: Erik Jaegervall <erik.jaegervall@se.bosch.com>
@gunnarx
Copy link
Collaborator Author

gunnarx commented Oct 31, 2022

Corrected the setup.py file according to the remarks noted above.

Gunnar Andersson and others added 5 commits October 31, 2022 19:22
@gunnarx
parser: complain about unknown/misspelled keys
cd63ba8 
This configures `dacite` to throw an exception if the input (yaml-dict)
contains any unexpected data items (usually coming from a misspelled
or misplaced keyword in the YAML input).  Then we catch the exception
and output a message that is as useful as we can, given the information.

Signed-off-by: Gunnar Andersson <gunnar_dev@nospam@novaspring.eu>
@gunnarx
tests: Support subdirs for each test
b42e433 
- Each test is now stored in a subdir under tests.
- Each test subdir must be named test.<something>.
- gen_test.py loops over those directories and performs the same test as
  before (generate the given 'input.py' using 'template' and compare to
  'result')

Signed-off-by: Gunnar Andersson <gunnar@nospam@novaspring.eu>
@gunnarx
Add test for illegal keys
df2051d 
The test only checks that the expected exception is raised on illegal
input.

Signed-off-by: Gunnar Andersson <gunnar@nospam@novaspring.eu>
@gunnarx
GitHub: Change python 3.10.8 -
44f38a1 
It seems that GitHub bumps the patch level number frequently.
(right now from 3.10.7 to 3.10.8)
Let's go with 3.10 instead of the more specific 3.10.7 to see
if this works more long term.
@gunnarx
Add a README inside tests for developer information
011c1a8
@gunnarx gunnarx mentioned this pull request Nov 7, 2022
Open
@magnusfeuer-mbition
Initial checkin of vsclib code to programatically build up and verify…
f4dbe69 
… a VSC tree

Signed-off-by: Magnus Feuer <magnus.feuer@mbition.io>
vsc_lib: spelling fixes
1d96955 
Simple fixes according to PR feedback - can be squashed during merge.
vsc_lib binary->byteBuffer
97dba61 
byteBuffer is the name used in the current specification
vsc_lib: Change exit code
4bc0e10 
(minor, can be squashed during merge)
vsclib/setup.py: Correct URL
26d98f7
vsclib/setup.py: Use python 3.10
5b4d49f
vsclib/setup.py: Change license information.
8ef8282 
It it self-evident from the LICENSE file that this was intended to be
MPLv2 licensed and the setup.py script should reflect this.

(When it becomes part of vsc-tools, then this setup.py might even be
 superfluous and therefore removed)
@gunnarx
Copy link
Collaborator Author

gunnarx commented Nov 14, 2022

Just rebased on master as a starting point, because there were 34 new commits.

@gunnar-mb
Copy link
Collaborator

gunnar-mb commented Jan 23, 2024 •
edited

Reference: #100 implements a more rudimentary version of this support for creating IFEX (previously VSC) definition from within a python program. The constructors provided by #100 are derived directly from the current IFEX language model.

The original vsc_lib does a bit more by including code that does type resolution, checking if types are visible through the namespace visibility rules, etc.

However, at least the construction of an IFEX interface tree representation is likely covered adequately by #100 now, with the advantage of it being up to date (and automatically up to date when/if the model changes).

@gunnar-mb gunnar-mb changed the title Vsc lib with fixes Long-term draft topic: Vsc lib with fixes Jan 23, 2024
@gunnar-mb gunnar-mb deleted the branch COVESA:vsclib June 4, 2024 15:43
@gunnar-mb gunnar-mb closed this Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erikbosch erikbosch erikbosch left review comments

Assignees

@gunnarx gunnarx

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@gunnarx @gunnar-mb @erikbosch @miketsukerman @magnusfeuer-mbition