Error when running mbed test --compile/run list

[MarceloSalazar]

During OOB for 5.10 release, found issues while evaluating icetea:

Mbed CLI 1.8.0
Mbed OS (920db63)
Windows 10

mbed test -m K64F -t ARM --icetea --compile-list -vv
...

[ERROR] Error when running icetea.
cwd:C:\mbed\OOB
Command:'icetea --tcdir TEST_APPS --list --json --platform_filter K64F -v'
output:Traceback (most recent call last):
  File "C:\Python27\Scripts\icetea-script.py", line 6, in <module>
    from pkg_resources import load_entry_point
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 3017, in <module>
    @_call_aside
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 3003, in _call_aside
    f(*args, **kwargs)
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 3030, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 661, in _build_master
    return cls._build_from_requirements(__requires__)
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 674, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 853, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'mbed-ls==1.*,>=1.5.1' distribution was not found and is required by icetea, mbed-flasher

[mbed] ERROR: "c:\python27\python.exe" returned error.
       Code: 1
       Path: "C:\mbed\OOB"
       Command: "c:\python27\python.exe -u C:\mbed\OOB\mbed-os\tools\run_icetea.py -m K64F -t ARM -v --compile-list"
       Tip: You could retry the last command with "-v" flag for verbose output

mbed test -m K64F -t ARM --icetea --run-list -vv
...

[ERROR] Error when running icetea.
cwd:C:\mbed\OOB
Command:'icetea --tcdir TEST_APPS --list --json --platform_filter K64F -v'
output:Traceback (most recent call last):
  File "C:\Python27\Scripts\icetea-script.py", line 6, in <module>
    from pkg_resources import load_entry_point
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 3017, in <module>
    @_call_aside
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 3003, in _call_aside
    f(*args, **kwargs)
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 3030, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 661, in _build_master
    return cls._build_from_requirements(__requires__)
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 674, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "c:\python27\lib\site-packages\pkg_resources\__init__.py", line 853, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'mbed-ls==1.*,>=1.5.1' distribution was not found and is required by icetea, mbed-flasher

[mbed] ERROR: "c:\python27\python.exe" returned error.
       Code: 1
       Path: "C:\mbed\OOB"
       Command: "c:\python27\python.exe -u C:\mbed\OOB\mbed-os\tools\run_icetea.py -m K64F -t ARM -v --build-data ..\BUILD\tests\K64F\ARM\build_data.json --test-suite ..\BUILD\tests\K64F\ARM\test_suite.json --run-list"
       Tip: You could retry the last command with "-v" flag for verbose output

The python traceback error should be caught appropriately with the corresponding message to the user

[OPpuolitaival]

Fix: #8075
[Mirrored to Jira]

[MarceloSalazar]

@OPpuolitaival thanks for the PR

However, I still don't think it's enough. The dependencies should be checked first, and traceback error should be caught.
Maybe this is something that @theotherjimmy can help?
[Mirrored to Jira]

[MarceloSalazar]

Manually upgraded mbed-ls

>mbedls --version
1.6.0

Problem continues

C:\mbed\OOB>mbed test -m K64F -t ARM --icetea --compile-list -vv
[mbed] Working path "C:\mbed\OOB" (program)
[mbed] Exec "c:\python27\python.exe -m pip list -l" in "C:\mbed\OOB"
Package                       Version
----------------------------- -----------
appdirs                       1.4.3
asn1ate                       0.5
asn1crypto                    0.22.0
beautifulsoup4                4.5.3
certifi                       2017.11.5
cffi                          1.10.0
chardet                       3.0.4
colorama                      0.3.9
cryptography                  2.3.1
ecdsa                         0.13
enum34                        1.1.6
fasteners                     0.14.1
funcsigs                      1.0.2
functools32                   3.2.3.post2
future                        0.16.0
futures                       3.2.0
fuzzywuzzy                    0.15.0
gcovr                         4.1
icetea                        1.0.1
idna                          2.6
intelhex                      2.1
ipaddress                     1.0.18
Jinja2                        2.9.5
jsonmerge                     1.5.1
jsonschema                    2.6.0
junit-xml                     1.8
lockfile                      0.12.2
Logbook                       1.4.0
lxml                          4.2.4
manifest-tool                 1.4.5
MarkupSafe                    0.23
mbed-cli                      1.8.0
mbed-cloud-sdk                2.0.0
mbed-flasher                  0.9.0
mbed-greentea                 1.3.3
mbed-host-tests               1.2.0
mbed-ls                       1.6.0
mock                          2.0.0
monotonic                     1.4
packaging                     16.8
pbr                           3.1.1
pip                           18.0
prettytable                   0.7.2
project-generator             0.9.10
project-generator-definitions 0.2.34
protobuf                      3.5.2.post1
py                            1.6.0
pyasn1                        0.2.3
pycparser                     2.17
pyelftools                    0.24
pyOCD                         0.8.1a1
pyparsing                     2.2.0
pyserial                      3.4
pyshark-legacy                0.3.8
python-dateutil               2.7.3
python-dotenv                 0.9.1
pywinusb                      0.4.2
PyYAML                        3.12
requests                      2.18.4
semver                        2.8.1
setuptools                    28.8.0
six                           1.11.0
trollius                      1.0.4
urllib3                       1.22
websocket-client              0.44.0
xmltodict                     0.10.2
yattag                        1.10.0
[mbed] Auto-installing missing Python modules...
[mbed] Exec "c:\python27\python.exe -m pip install -q -r C:\mbed\OOB\mbed-os\requirements.txt" in "C:\mbed\OOB"

[mbed] Exec "c:\python27\python.exe -u C:\mbed\OOB\mbed-os\tools\run_icetea.py -m K64F -t ARM -v --compile-list" in "C:\mbed\OOB"
Traceback (most recent call last):
  File "C:\mbed\OOB\mbed-os\tools\run_icetea.py", line 268, in <module>
    icetea_json_output = icetea_tests(options.target, options.tcdir, options.verbose)
  File "C:\mbed\OOB\mbed-os\tools\run_icetea.py", line 167, in icetea_tests
    stderr.decode()))
Exception: Error when running icetea.
cwd:C:\mbed\OOB
Command:'icetea --tcdir TEST_APPS --list --json --platform_filter K64F -v'
output:11:47:34.355 Registering execution wide plugins:
11:47:34.355 Registering plugin LocalAllocator
11:47:34.355 Execution wide plugins loaded and registered.
11:47:34.371 Using Icetea version 1.0.1
11:47:34.371 Generating suite from command line.
11:47:34.371 Importing testcases for filtering
11:47:34.371 Test case directory does not exist!
11:47:34.371 Something went wrong in suite creation! Suite creation failed.

[ERROR] Error when running icetea.
cwd:C:\mbed\OOB
Command:'icetea --tcdir TEST_APPS --list --json --platform_filter K64F -v'
output:11:47:34.355 Registering execution wide plugins:
11:47:34.355 Registering plugin LocalAllocator
11:47:34.355 Execution wide plugins loaded and registered.
11:47:34.371 Using Icetea version 1.0.1
11:47:34.371 Generating suite from command line.
11:47:34.371 Importing testcases for filtering
11:47:34.371 Test case directory does not exist!
11:47:34.371 Something went wrong in suite creation! Suite creation failed.

[mbed] ERROR: "c:\python27\python.exe" returned error.
       Code: 1
       Path: "C:\mbed\OOB"
       Command: "c:\python27\python.exe -u C:\mbed\OOB\mbed-os\tools\run_icetea.py -m K64F -t ARM -v --compile-list"
       Tip: You could retry the last command with "-v" flag for verbose output

[Mirrored to Jira]

[theotherjimmy]

@MarceloSalazar The requirements are checked before running any commands.
[Mirrored to Jira]

[MarceloSalazar]

The requirements might be checked, but I still get traceback errors, which are not nice!

What is this Code 1 ?

[mbed] ERROR: "c:\python27\python.exe" returned error.
       Code: 1

Don't know how to proceed
[Mirrored to Jira]

[MarceloSalazar]

#8075 is NOT fixing the problem
[Mirrored to Jira]

[OPpuolitaival]

Here is two problems:

1. Wrong lib numbers which are fixed in Syncronize mbed-ls minimum version with icetea requirements #8075
   @theotherjimmy pip does not check sub-debendencies and therefore some library may need greater version of something than Mbed OS and only Mbed OS libs are checked

2. Root cause problem. The missing feature which is described better way in Icetea must be run from the mbed-os root #8120
   [Mirrored to Jira]

[cmonr]

Looks like #8075 is now merged.

This now depends on #8120 and can then be closed.
[Mirrored to Jira]

[0xc0170]

This now depends on #8120 and can then be closed.

Can this be closed now?
[Mirrored to Jira]

[MarceloSalazar]

Sorry I still can't get this to work with RC3
I'm still seeing the error:

stderr.decode()))
Exception: Error when running icetea.

Even tried moving to mbed-os as suggested in #8120
[Mirrored to Jira]

[jupe]

is above error same as #8172 ?
[Mirrored to Jira]

[OPpuolitaival]

The root cause is "11:47:34.371 Test case directory does not exist!" which tells that there is no test case folder available. The folder name in this case is "TEST_APPS". That folder need to exist in path where you run icetea tests. This is fixed in mbed-cls but not released yet. There is PR made in mbed-os also which makes error code a bit better #8181
[Mirrored to Jira]

[MarceloSalazar]

I've updated to Mbed CLI 1.8.2.

1. When running the command in the root directory, see this message:

    mbed test -m K64F -t GCC_ARM --icetea --compile-list
   ...
   [mbed] WARNING: Cannot run icetea tests. Current folder does not contain TEST_APPS folder.
   

The tools should be able to look into internal folders! But the warning message is ok for now.

2. Running the command inside mbed-os, I still see the error stderr.decode()))

EDIT:
I'm seeing this problem in both OSX+Python 3.7.0 , and Win10+Python2.7.15
I do have a TEST_APPS inside Mbed OS (5.10-rc3)

EDIT2:
The problem doesn't seem to happen when using Mbed OS master

[Mirrored to Jira]

[jamesbeyond]

I did some experiment with it. seems this has something to do with how mbed-os project structure and how mbed tools invoked.
CASE 1:

git clone https://github.com/ARMmbed/mbed-os.git  
cd mbed-os
git checkout mbed-os-5.10.0-rc3
mbed test -m K64F -t GCC_ARM --icetea --compile-list

CASE 2:

mbed new foo
cd foo\mbed-os
mbed update mbed-os-5.10.0-rc3
mbed test -m K64F -t GCC_ARM --icetea --compile-list

CASE1 worked with icetea, CASE2 not

Seems the mbed tools treat CASE2 as a mbed application.
So if you remove the mbed-os.lib or mbed_settings.py in the foo folder make mbed tools think the app is broken, then icetea works again
[Mirrored to Jira]

[screamerbg]

CC @cmonr @theotherjimmy @OPpuolitaival
[Mirrored to Jira]

[cmonr]

Looking at this now. It appears that when invoking test case #2, the root directory is modified to be the foo directory, instead of mbed-os.
[Mirrored to Jira]

[MarceloSalazar]

IMO the tools should be able to run icetea regardless of where you have the root directory configured.
This is the standard behavior with greentea and we should work on getting icetea working the same way.

[Mirrored to Jira]

[MarceloSalazar]

FYI @maclobdell @bentcooke
[Mirrored to Jira]

[cmonr]

After talking with a couple of people who are more familiar with the intended legacy behavior of mbed test, I think this is an issue with icetea instead of mbed-cli.

We want to enable and encourage application developers to develop their own tests, and other classes of tests should be able to support this behavior. Going to pause investigaton into root causing where the change in directory is occuring.
[Mirrored to Jira]

[jupe]

I think this is an issue with icetea instead of mbed-cli.

@cmonr could you open this a bit more, is there something we could improve in icetea side or is this related only to icetea integration work ?
[Mirrored to Jira]

[cmonr]

@jupe I think this was a mistake with the initial scope of enabling icetea t0 be runnable via mbed test. I don't think it was communicated anywhere that test in mbed-os should also be runable if the command is invoked from a project.

On the icetea side of things, it would be useful if it could handle being invoked from a user's project directory, not just within the mbed-os directory. The thinking goes as such: if a user wants to add tests specific to their application, they'd add it inside of their project, not within mbed-os. We initially had greentea tests, but now have unittests and icetea tests, so both sets of new test should be able to support this behavior.

Does that make sense?
[Mirrored to Jira]

[jupe]

Icetea point of view there is no any restrictions like that. Icetea itself are standalone application that does not depend on mbed-os neither mbed-cli. Probably only limitations related to how integration to mbed-os/cli are implemented - there are currently some assumptions like what you saw, but I’m pretty sure that if requirements are clear @OPpuolitaival can improve current implementation to fulfill this requirement as well.

edit: is it actually this what you are explaining: #8120 ?
[Mirrored to Jira]

[OPpuolitaival]

=> MBEDOSTEST-200
[Mirrored to Jira]

[adbridge]

Internal Jira reference: https://jira.arm.com/browse/IOTCORE-437