Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update examples testing scripts #11710

Merged
merged 15 commits into from Oct 28, 2019
Merged

Update examples testing scripts #11710

merged 15 commits into from Oct 28, 2019

Conversation

@jamesbeyond
Copy link
Contributor

jamesbeyond commented Oct 18, 2019

Description

This PR is update the examples testing scripts.
This script is standalone script purely used by mbed CI test, So this will not change mbed-OS behaviours.

This including following changes:

  • Add a README file to explain how this script is used in the CI
  • Removed the dependency of Mercurial, This script will only use Github for check out code
  • Add a function for the script to create symbolic links to mbed-os.
  • Add a logging tool, allow the STDOUT to be easy to debug
  • Update the output message, print a nicer result summary

However, because this scripts changed where to checkout examples, some of the examples will failed to build

  • mbed-os-example-BLE
  • mbed-os-example-tls
  • mbed-os-example-crypto

So temporarily turn off the build tests for these examples. An update on the CI scripts will be raised soon. Once the CI update is in place will enable the build test for these examples ASAP.

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[x] Test update
[ ] Breaking change

Reviewers

@OPpuolitaival @evedon @adbridge @mark-edgeworth @AnotherButler

Release Notes

@ciarmcom ciarmcom requested review from adbridge, AnotherButler, evedon, mark-edgeworth, OPpuolitaival and ARMmbed/mbed-os-maintainers Oct 18, 2019
@ciarmcom

This comment has been minimized.

Copy link
Member

ciarmcom commented Oct 18, 2019

Copy link
Contributor

OPpuolitaival left a comment

Looks great!

* **name** - name of the example, should be the base name of the example repository address, will throw if not match
* **github** - example github repository address
* **sub-repo-example** specify if the example repository have sub folder for each examples
* **subs** - array of sub examples names

This comment has been minimized.

Copy link
@OPpuolitaival

OPpuolitaival Oct 18, 2019

Contributor

Which expected to be as folders in root of the repository?

This comment has been minimized.

Copy link
@jamesbeyond

jamesbeyond Oct 18, 2019

Author Contributor

Yes, as above TLS example, it just going to be benchmark tls-client hashing and authcrypt

Edit file, mostly for formatting and grammar.
@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

jamesbeyond commented Oct 20, 2019

Thanks for your updates on the README @AnotherButler

Copy link
Member

0xc0170 left a comment

"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-tls-client",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-hashing",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-authcrypt"
"sub-repo-example": true,

This comment has been minimized.

Copy link
@0xc0170

0xc0170 Oct 21, 2019

Member

Nice to have finally support for these. Can this be automated? If this is true, we walk through the root and if mbed-os.json file found, it is an example for testing ? This way we do not need to define subs (this can be out of date easily as everytime someone adds a new sub example or restructure example, this breaks).

Or any other way to do this without having this "subs" in here?

This comment has been minimized.

Copy link
@adbridge

adbridge Oct 22, 2019

Contributor

Removing these lines breaks the release script. I will need to consider the impact before the next release.

This comment has been minimized.

Copy link
@adbridge

adbridge Oct 23, 2019

Contributor

Nice to have finally support for these. Can this be automated? If this is true, we walk through the root and if mbed-os.json file found, it is an example for testing ? This way we do not need to define subs (this can be out of date easily as everytime someone adds a new sub example or restructure example, this breaks).

Or any other way to do this without having this "subs" in here?

@0xc0170 That is how the release script currently updates the mbed-os.lib files. It works down the root.

@jamesbeyond Actually looking at the release script more carefully I think this change is ok. The script only looks at the 'name' field so as long as that doesn't change we should be ok!

@0xc0170

This comment has been minimized.

Copy link
Member

0xc0170 commented Oct 21, 2019

So temporarily turn off the build tests for these examples. An update on the CI scripts will be raised soon. Once the CI update is in place will enable the build test for these examples ASAP.

What is the timeframe for this?

@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

jamesbeyond commented Oct 21, 2019

What is the timeframe for this?

Those tests should be turned back on within a week after this PR been merged

The scripts in this folder are used for testing `mbed-os` official examples. It contains the following files:

- `examples.py` - the main script that serves as the command-line interface.
- `examples.json` - the default configuration file for the script, which contains the information to the examples. If required, you can pass a customized configuration file as an argument to the script.

This comment has been minimized.

Copy link
@evedon

evedon Oct 22, 2019

Contributor

What do you mean by a customized configuration file? Can you show an example on how to use this feature?

I think that we currently don't have a way to test different profiles (bare metal profile/minimal-printf profile) and/or library configurations, do we? Is this PR the first step towards achieving this?

This comment has been minimized.

Copy link
@jamesbeyond

jamesbeyond Oct 22, 2019

Author Contributor

the json template in the README is an example of configuration. This means if other people want to use this in their own CI, they can do it like such. But regarding to mbed-OS main CI, it always uses the default configuration file.

The new scripts actually allow the minimal-printf to be tested, you can use:

python mbed-os/tools/test/examples/examples.py compile GCC_ARM -m <target> --profile develop mbed-os/tools/profile/extensions/minimal-printf.json
Number of failures = 0
```

After the compilation stage, a `test_spec.json` file is generated. Later, Greentea tests will consume this file. They test the compiled example on hardware platform.

This comment has been minimized.

Copy link
@evedon

evedon Oct 22, 2019

Contributor

Can this test_spec.json also include some specific test configuration settings, for example if we want to test bare metal?

This comment has been minimized.

Copy link
@jamesbeyond

jamesbeyond Oct 22, 2019

Author Contributor

this test_spec.json is generated after the compilation is done. it contains the if where is the binary file, where is the compare_log. in terms of bare-metal profile it need to be passed in to the compilation as mbed_apps.json. If we have the baremetal example, it can test the baremetal example without any issues

@adbridge

This comment has been minimized.

Copy link
Contributor

adbridge commented Oct 22, 2019

General comment. I'm not happy about turning off ci testing of the 3 sets of examples mentioned! These are not really tested anywhere else so this is a dangerous thing to do....

@adbridge

This comment has been minimized.

Copy link
Contributor

adbridge commented Oct 22, 2019

Also the changes you have made to the examples with multiple sub examples will now break the release script. Examples.json is parsed by the release script to work out which examples to update the mbed-os.lib files for....

"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-tls-client",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-hashing",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-authcrypt"
"sub-repo-example": true,

This comment has been minimized.

Copy link
@adbridge

adbridge Oct 22, 2019

Contributor

Removing these lines breaks the release script. I will need to consider the impact before the next release.

@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

jamesbeyond commented Oct 22, 2019

General comment. I'm not happy about turning off ci testing of the 3 sets of examples mentioned! These are not really tested anywhere else so this is a dangerous thing to do....

Maybe I didn't made myself clear enough, it just temporary turned off. very soon they will be added back to the compile test. the reason is I need to make any other change to the CI groovy script, once the CI groovy script is done I can re-enable them in mbed-os. then the CI will check those example out from Github instead of Mercurial.

Removing these lines breaks the release script. I will need to consider the impact before the next release.

I wasn't aware of the impacts to the release scripts, let have a catch-up offline regarding this issue. @adbridge

"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-tls-client",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-hashing",
"https://os.mbed.com/teams/mbed-os-examples/code/mbed-os-example-tls-authcrypt"
"sub-repo-example": true,

This comment has been minimized.

Copy link
@adbridge

adbridge Oct 23, 2019

Contributor

Nice to have finally support for these. Can this be automated? If this is true, we walk through the root and if mbed-os.json file found, it is an example for testing ? This way we do not need to define subs (this can be out of date easily as everytime someone adds a new sub example or restructure example, this breaks).

Or any other way to do this without having this "subs" in here?

@0xc0170 That is how the release script currently updates the mbed-os.lib files. It works down the root.

@jamesbeyond Actually looking at the release script more carefully I think this change is ok. The script only looks at the 'name' field so as long as that doesn't change we should be ok!

Copy link
Member

0xc0170 left a comment

Although less error prone would be : if subs is not empty use it as "subs repo example" otherwise a repo is just one. This would be simplified: https://github.com/ARMmbed/mbed-os/pull/11710/files#diff-e373941cca74280be2386207895bc63aR167

@0xc0170

This comment has been minimized.

Copy link
Member

0xc0170 commented Oct 25, 2019

CI started

@0xc0170 0xc0170 added needs: CI and removed needs: review labels Oct 25, 2019
@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

jamesbeyond commented Oct 25, 2019

Although less error prone would be : if subs is not empty use it as "subs repo example" otherwise a repo is just one. This would be simplified: https://github.com/ARMmbed/mbed-os/pull/11710/files#diff-e373941cca74280be2386207895bc63aR167

Yeah, I thought about that, but I think the empty value [ ], seems misleading sometimes. Because in other fields, the [ ] means all possible alternatives rather than nothing. So just want to have this explicit filed that shows this example dose contains sub-examples

@mbed-ci

This comment has been minimized.

Copy link

mbed-ci commented Oct 25, 2019

Test run: FAILED

Summary: 3 of 4 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_build-ARM
  • jenkins-ci/mbed-os-ci_build-GCC_ARM
  • jenkins-ci/mbed-os-ci_build-IAR
@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

jamesbeyond commented Oct 25, 2019

Sorry, forgot to turn-off the build test for NFC example, they are with sub-examples as well. Now updated, Could we start the CI again please? @0xc0170

@mbed-ci

This comment has been minimized.

Copy link

mbed-ci commented Oct 28, 2019

Test run: SUCCESS

Summary: 11 of 11 test jobs passed
Build number : 2
Build artifacts

@0xc0170 0xc0170 added ready for merge and removed needs: CI labels Oct 28, 2019
@0xc0170 0xc0170 merged commit 83c4a80 into ARMmbed:master Oct 28, 2019
26 checks passed
26 checks passed
continuous-integration/jenkins/pr-head This commit looks good
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
jenkins-ci/build-ARM Success
Details
jenkins-ci/build-GCC_ARM Success
Details
jenkins-ci/build-IAR Success
Details
jenkins-ci/cloud-client-test Success
Details
jenkins-ci/dynamic-memory-usage RTOS ROM(+0 bytes) RAM(-72 bytes)
Details
jenkins-ci/exporter Success
Details
jenkins-ci/greentea-test Success
Details
jenkins-ci/mbed2-build-ARM Success
Details
jenkins-ci/mbed2-build-GCC_ARM Success
Details
jenkins-ci/mbed2-build-IAR Success
Details
jenkins-ci/unittests Success
Details
travis-ci/astyle Success!
Details
travis-ci/docs Success!
Details
travis-ci/doxy-spellcheck Success!
Details
travis-ci/events Success! Runtime is 8693 cycles.
Details
travis-ci/gitattributestest Success!
Details
travis-ci/include_check Success!
Details
travis-ci/licence_check Success!
Details
travis-ci/littlefs Success! Code size is 8420B.
Details
travis-ci/psa-autogen Success!
Details
travis-ci/tools-py2.7 Success!
Details
travis-ci/tools-py3.5 Success!
Details
travis-ci/tools-py3.6 Success!
Details
travis-ci/tools-py3.7 Success!
Details
@0xc0170

This comment has been minimized.

Copy link
Member

0xc0170 commented Oct 28, 2019

@jamesbeyond Integrated, lets update CI

devran01 added a commit to devran01/mbed-os that referenced this pull request Oct 30, 2019
devran01 added a commit to devran01/mbed-os that referenced this pull request Oct 30, 2019
devran01 added a commit to devran01/mbed-os that referenced this pull request Oct 31, 2019
Update examples testing scripts

(cherry picked from commit 83c4a80)
@jamesbeyond jamesbeyond deleted the jamesbeyond:exp_github branch Oct 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.