Specifying PYTHONPATH for contrib builds #361

erikbosch · 2021-11-10T15:12:23Z

Background:

If not set the local vspec will not be first candidate,
as it is not located under contrib.
Instead a cached version might be used, which might not be compatible
with the current vss-tools version

Background: If not set the local vspec will not be first candidate, as it is not located under contrib. Instead a cached version might be used, which might not be compatible with the current vss-tools version

erikbosch · 2021-11-10T15:21:30Z

@danielwilms - I believe this solves the issue we have seen recently on failing contrib-builds in CI.

More info on background:

We noticed that many contrib builds failed (even those expected to pass). Daniel noticed that a cached vss-tools vspec module seemed to be used. I noticed that it seemed to be an error sometimes appearing, sometimes not. This is likely related to the recent introduction of config.yaml. It seems that we sometimes when building vss fetch a newer cached vspec module which expect config.yaml to exist, but it does not.

With my limited knowledge it seems Python prefer a cached vspec module rather than something located in ../vspec. Setting PYTHONPATH seems to fix this.

erikbosch · 2021-11-15T07:53:50Z

More details on the problem, showing two recent builds.

https://github.com/COVESA/vehicle_signal_specification/runs/4176114945?check_suite_focus=true

This build works as expected, there are problems building vspec2c and vspec2ocf due to that those tools have not been updated for a while, but the errors are as expected.


Run make -k travis_optional || true
rm -f vss_rel_*
(cd ./vss-tools/contrib/vspec2c/; make clean)
make[1]: Entering directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
rm -f vehicle_signal_specification.o libvss.a libvss.so *~
make[2]: Entering directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c/example'
rm -f vss_dump.o vss_dump signal_spec.h signal_macro.h *~
make[2]: Leaving directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c/example'
make[1]: Leaving directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
(cd ./vss-tools/contrib/vspec2c/; make )
make[1]: Entering directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
cc -fPIC -O2 -Wall  -I.   -c -o vehicle_signal_specification.o vehicle_signal_specification.c
ar r libvss.a vehicle_signal_specification.o
ar: creating libvss.a
cc -shared -fPIC -O2 -Wall  -I. -o libvss.so vehicle_signal_specification.o
make[1]: Leaving directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
PYTHONPATH=./vss-tools ./vss-tools/contrib/vspec2c.py -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).h vss_rel_$(cat VERSION)_macro.h
./vss-tools/contrib/vspec2c.py:296: UnsupportedWarning: load is unsupported as of 2.0. Anytree as tree library introduced for typesafe handling of vss structure.
Illegal data type: 'uint8[]'
  tree = vspec.load(args[0], include_dirs)
Signal: Vehicle_Cabin_SeatPosCount
Try: int8 uint8 int16 uint16 int32 uint32 int64 uint64
     float double string boolean stream
make: *** [Makefile:60: c] Error 255
./vss-tools/contrib/ocf/vspec2ocf.py -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).ocf.json
./vss-tools/contrib/ocf/vspec2ocf.py:165: UnsupportedWarning: load is unsupported as of 2.0. Anytree as tree library introduced for typesafe handling of vss structure.
  tree = vspec.load(args[0], include_dirs)
Traceback (most recent call last):
  File "./vss-tools/contrib/ocf/vspec2ocf.py", line 174, in <module>
    for key,value in data.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'
make: *** [Makefile:53: ocf] Error 1
./vss-tools/contrib/vspec2protobuf.py  -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).proto
./vss-tools/contrib/vspec2ttl/vspec2ttl.py -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).ttl
Traceback (most recent call last):
  File "./vss-tools/contrib/vspec2ttl/vspec2ttl.py", line 29, in <module>
    from rdflib import Graph, Literal, RDF, URIRef, Namespace
ModuleNotFoundError: No module named 'rdflib'
make: *** [Makefile:56: ttl] Error 1
./vss-tools/contrib/vspec2graphql.py -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).graphql.ts
WARNING: Skipping branch with no children: Vehicle.Private
WARNING: Skipping branch with no children: Vehicle.Media
WARNING: Skipping branch with no children: Vehicle.Powertrain.FuelCell
make: Target 'travis_optional' not remade because of errors.

https://github.com/COVESA/vehicle_signal_specification/runs/4163571834?check_suite_focus=true

In this build the problem is visible. It complains on that vspec/config.yaml does not exist, But it should not, as that code exists in a newer version of vss-tools that not yet is used by this version of vss. One can see from the stacktrace that it uses a cached version of vss-tools (in /opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/vss_tools-d5a2e70f-py3.8.egg/vspec/model/vsstree.py) instead of the local one.

Run make -k travis_optional || true
rm -f vss_rel_*
(cd ./vss-tools/contrib/vspec2c/; make clean)
make[1]: Entering directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
rm -f vehicle_signal_specification.o libvss.a libvss.so *~
make[2]: Entering directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c/example'
rm -f vss_dump.o vss_dump signal_spec.h signal_macro.h *~
make[2]: Leaving directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c/example'
make[1]: Leaving directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
(cd ./vss-tools/contrib/vspec2c/; make )
make[1]: Entering directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
cc -fPIC -O2 -Wall  -I.   -c -o vehicle_signal_specification.o vehicle_signal_specification.c
ar r libvss.a vehicle_signal_specification.o
ar: creating libvss.a
cc -shared -fPIC -O2 -Wall  -I. -o libvss.so vehicle_signal_specification.o
make[1]: Leaving directory '/home/runner/work/vehicle_signal_specification/vehicle_signal_specification/vss-tools/contrib/vspec2c'
PYTHONPATH=./vss-tools ./vss-tools/contrib/vspec2c.py -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).h vss_rel_$(cat VERSION)_macro.h
./vss-tools/contrib/vspec2c.py:296: UnsupportedWarning: load is unsupported as of 2.0. Anytree as tree library introduced for typesafe handling of vss structure.
Illegal data type: 'uint8[]'
  tree = vspec.load(args[0], include_dirs)
Signal: Vehicle_Cabin_SeatPosCount
Try: int8 uint8 int16 uint16 int32 uint32 int64 uint64
     float double string boolean stream
make: *** [Makefile:60: c] Error 255
./vss-tools/contrib/ocf/vspec2ocf.py -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).ocf.json
Traceback (most recent call last):
  File "<frozen zipimport>", line 177, in get_data
KeyError: 'vspec/config.yaml'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./vss-tools/contrib/ocf/vspec2ocf.py", line 25, in <module>
    import vspec
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/vss_tools-d5a2e70f-py3.8.egg/vspec/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/vss_tools-d5a2e70f-py3.8.egg/vspec/model/vsstree.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/vss_tools-d5a2e70f-py3.8.egg/vspec/model/constants.py", line 186, in <module>
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/vss_tools-d5a2e70f-py3.8.egg/vspec/model/constants.py", line 187, in Unit
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/vss_tools-d5a2e70f-py3.8.egg/vspec/model/constants.py", line 63, in get_members_from_default_config
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1136, in resource_stream
    return get_provider(package_or_requirement).get_resource_stream(
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1385, in get_resource_stream
    return io.BytesIO(self.get_resource_string(manager, resource_name))
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1388, in get_resource_string
    return self._get(self._fn(self.module_path, resource_name))
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1555, in _get
    return self.loader.get_data(path)
  File "<frozen zipimport>", line 179, in get_data
OSError: [Errno 0] : 'vspec/config.yaml'
make: *** [Makefile:53: ocf] Error 1
./vss-tools/contrib/vspec2protobuf.py  -I ./spec ./spec/VehicleSignalSpecification.vspec vss_rel_$(cat VERSION).proto
Traceback (most recent call last):
  File "<frozen zipimport>", line 177, in get_data
KeyError: 'vspec/config.yaml'

danielwilms · 2021-11-16T12:23:01Z

Great that it fixes the build for now. Wondering if we should extend pythonpath instead of overwriting.

Then we should consider to implement COVESA/vss-tools#114, which should solve this one as well I'd say.

erikbosch · 2021-11-16T15:00:10Z

Great that it fixes the build for now. Wondering if we should extend pythonpath instead of overwriting.

Then we should consider to implement COVESA/vss-tools#114, which should solve this one as well I'd say.

You mean something like below?

PYTHONPATH=${TOOLSDIR}:${PYTHONPATH}

danielwilms · 2021-11-16T18:11:39Z

@oliveirarleo please have a look, if it's referenced correctly in python scripts

oliveirarleo · 2021-11-16T18:39:35Z

Turns out that putting TOOLSDIR on PYTHONPATH would be a conceptual error. The problem was that at some moment I removed a modification on setup.py where I specify where vspec/config.yaml should be included.

oliveirarleo · 2021-11-16T18:48:42Z

The PR at vss-tools #119 solves this issue.

erikbosch · 2021-11-17T07:54:15Z

The PR at vss-tools #119 solves this issue.

Seems to be correct, I will close this PR. Thanks for the help and analysis @oliveirarleo

Specifying PYTHONPATH for contrib builds

aea0b9a

Background: If not set the local vspec will not be first candidate, as it is not located under contrib. Instead a cached version might be used, which might not be compatible with the current vss-tools version

erikbosch marked this pull request as ready for review November 10, 2021 15:14

erikbosch closed this Nov 17, 2021

erikbosch deleted the erik_build branch January 10, 2022 13:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Specifying PYTHONPATH for contrib builds #361

Specifying PYTHONPATH for contrib builds #361

erikbosch commented Nov 10, 2021

erikbosch commented Nov 10, 2021

erikbosch commented Nov 15, 2021

danielwilms commented Nov 16, 2021

erikbosch commented Nov 16, 2021

danielwilms commented Nov 16, 2021

oliveirarleo commented Nov 16, 2021 •

edited

Loading

oliveirarleo commented Nov 16, 2021

erikbosch commented Nov 17, 2021

Specifying PYTHONPATH for contrib builds #361

Specifying PYTHONPATH for contrib builds #361

Conversation

erikbosch commented Nov 10, 2021

erikbosch commented Nov 10, 2021

erikbosch commented Nov 15, 2021

danielwilms commented Nov 16, 2021

erikbosch commented Nov 16, 2021

danielwilms commented Nov 16, 2021

oliveirarleo commented Nov 16, 2021 • edited Loading

oliveirarleo commented Nov 16, 2021

erikbosch commented Nov 17, 2021

oliveirarleo commented Nov 16, 2021 •

edited

Loading