Make update.py test compile examples prior to updating mbed-os version. #3048

adbridge · 2016-10-17T16:31:08Z

@bridadan please review ? :)

Changes: Refactor examples.py to add a new command line option to provide an update tag. Refactor examples.py to add new functionality to update the version of mbed-os in the examples to a supplied tag. Refactor examples.py to make cloning the example repos, updating their mbed-os version and compiling, into library functions and move to a new library module. Refactor the format of the examples.json file to make it compatible with both examples.py and update.py. Refactor update.py so that examples are test compiled prior to updating. Refactor update.py so that only examples tagged as auto-updatable and that fully compile are automatically updated.

bridadan

Minor comment about use of rm, but everything else looks ok to me!

bridadan · 2016-10-17T18:03:02Z

tools/test/examples/examples_lib.py

+            name = basename(repo)
+            if os.path.exists(name):
+                print("'%s' example directory already exists. Deleting..." % name)
+                subprocess.call(['rm', '-rf', name])


Here you're making the assumption that we're running in a POSIX shell. I'd prefer to use the built in Python commands for this (rmtree) if possible to keep this cross platform. Is there a good reason to only allow this to run in a POSIX shell?

bridadan · 2016-10-17T18:12:12Z

@theotherjimmy Could you take a brief look at this as well since you wrote the initial version of this?

bridadan · 2016-10-17T18:12:59Z

/morph test

mbed-bot · 2016-10-18T00:34:55Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: null

Examples Build failed!

adbridge · 2016-10-18T10:35:54Z

Looks like the examples program has run successfully - it has found some problems with the following example - mbed-os-example-tls-benchmark:

A number of boards have failed. Talking to the TLS guys there are only a limited number of boards that they expect the example to work on. This raises a question about how do we determine which boards to run the examples on . We have the ability to limit the test to a list of boards in the script so do we just take the decision to run across a subset of known , supported boards?

Suggestion is to limit tls to the following:
"targets" : ["K64F", "NUCLEO_F429ZI"],
"toolchains" : ["GCC_ARM", "ARM"],
I noticed that currently the tls examples weren't included in the json file whereas this change has brough them in .

Update examples.json to limit TLS examples to a couple of boards and only GCC_ARM and ARM compilers.

adbridge · 2016-10-18T11:32:26Z

/morph test

mbed-bot · 2016-10-18T15:14:20Z

Result: ABORTED

Your command has finished executing! Here's what you wrote!

/morph test

adbridge · 2016-10-18T17:23:03Z

To allow more configurability for which examples should be compiled and which shouldn't, add another field to the .json file :
"compile" : true/false

theotherjimmy

Minor niggles.

theotherjimmy · 2016-10-21T19:16:16Z

tools/test/examples/examples_lib.py

+
+       The results are returned in a [key: value] dictionary format:
+       Where key = The example name from the json config file
+             value = a list containing: pass_status, successes, and failures failures


Failures failures!?

What's a failure failure, A success?

theotherjimmy · 2016-10-21T19:21:48Z

tools/test/examples/update.py

+    for root, dirs, files in os.walk(path):
+        for file in files:
+            if file == 'mbed-os.lib':
+                examples += [root]


Should this be:

if 'mbed-os.lib' in files: examples += [root]

It's just as clear.

theotherjimmy · 2016-10-21T19:23:28Z

tools/test/examples/update.py

+        return False
+
+    os.chdir(cwd)
+    return True


return not return_code

then get rid of that silly if statement above.

theotherjimmy · 2016-10-21T19:24:26Z

tools/test/examples/update.py

+        return False
+
+    os.chdir(cwd)
+    return True


Same as above:

return not return_code

0xc0170 · 2016-10-24T16:17:49Z

@adbridge Can you please have a look at the comments?

adbridge · 2016-10-24T16:20:07Z

@0xc0170 being worked on currently

…file. Adding the new compile option allows the marking of a set of examples to indicate whether they should be compiled or not. For the update process examples that are not compiled will not be auto updated irrespective of that setting. Other changes to make return logic from some functions in update.py more efficient and some typos in the lib file.

adbridge · 2016-10-25T11:42:19Z

@theotherjimmy please can you check review updates and confirm happy.

bridadan

LGTM

bridadan · 2016-10-25T15:15:21Z

/morph test

mbed-bot · 2016-10-25T20:40:10Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: null

Examples Build failed!

bridadan · 2016-10-25T21:52:55Z

@adbridge looks like we need to filter out a few of these examples for the time being. Can you take a look?

adbridge · 2016-10-26T11:36:59Z

@bridadan I'm wondering if the wifi example needs a feature e.g. "WIFI" in much the same way as BLE has "BLE" ?
I've asked @pan- to comment on the BLE failures

…ons. Remove URI beacon example as this is no longer required. Add specific set of supported targets.

adbridge · 2016-10-26T15:31:00Z

/morph test

mbed-bot · 2016-10-26T17:47:03Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: null

Examples Build failed!

bridadan · 2016-10-27T14:50:35Z

@adbridge We can't merge this until the wifi example has been updated, otherwise all PRs will fail :) Though it looks like the PR against the wifi example was merged, so I'll rerun it now.

/morph test

mbed-bot · 2016-10-27T15:46:16Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: null

Examples Build failed!

bridadan · 2016-10-27T15:48:29Z

Ah, also #3139 has to be merged before this can be merged.

0xc0170 · 2016-10-28T08:59:54Z

Ah, also #3139 has to be merged before this can be merged.

Merged, any other dependency?

/morph test

adbridge · 2016-10-28T11:04:36Z

@0xc0170 If ci passes after the merge of #3139 then this will be good to go

mbed-bot · 2016-10-28T12:42:47Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: null

Examples Build failed!

adbridge · 2016-10-28T13:17:25Z

Failures are now genuine, transient failures due to :-
#3152
By default all examples should now compile successfully so this change is good to merge IMO.

0xc0170 · 2016-10-28T13:45:29Z

tools/test/examples/update.py

+    args = parser.parse_args(arguments)
+
+    cfg = os.path.join(os.path.dirname(__file__), args.config_file)
+    print cfg


what is this print, debug msg? Should be at least print(cfg)

Shouldn't be there! Well spotted and why did nobody else lol

0xc0170 · 2016-10-28T13:49:21Z

@bridadan The last set of eyes on this PR?

bridadan

@adbridge @0xc0170 We still can't merge this if it causes CI to fail, even if the fault lies with the mbed OS code base or an example (and not the update script itself). It will cause all other PRs to fail as well, which isn't useful :)

Instead, I would suggest turning off the compiling step for the WIFI example in this PR and get it merged. Then, when the issue in #3152 is fixed, all we have to do is submit a PR to turn on the compile flag again. OR better yet, the PR that fixes #3152 should turn on the compile flag so we can verify that the issue is actually fixed!

adbridge · 2016-10-28T14:56:54Z

Actually I think I would disagree. If we turn a whole suite of tests off while we await one fix then we stand to miss any other failures that may affect that suite of tests in the interim...Maybe we need someway of publishing known current failures ?

issue 3152 is submitted.

adbridge · 2016-10-28T15:13:51Z

/morph test

bridadan

Awesome, thanks for the changes @adbridge. LGTM after CI comes back

mbed-bot · 2016-10-28T19:05:26Z

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: 967

All builds and test passed!

sg- · 2016-11-01T15:28:54Z

@marhil01 @jupe feedback on pr-head CI?

Release mbed OS 5.2.2 and mbed lib v129 Known issues in this release There is currently a DNS resolution failure in Thread mode with this release. This causes a failure in the mbed-os-example-client. This will be fixed in a subsequent release. This can be worked around by reverting to mbed-os-5.2.0 Ports for Upcoming Targets 3011: Add u-blox Sara-N target. ARMmbed#3011 3099: MAX32625 ARMmbed#3099 3151: Add support for FRDM-K82F ARMmbed#3151 3177: New mcu k22512 fixing pr 3136 ARMmbed#3177 Fixes and Changes 2990: [tools] Parallel building of tests ARMmbed#2990 3008: NUCLEO_F072RB: Fix wrong timer channel number on pwm PB_5 pin ARMmbed#3008 3013: STM32xx - Change how the ADC internal pins are checked before pinmap_ ARMmbed#3013 3023: digital_loop tests update for STM32 ARMmbed#3023 3041: [nRF5] - added implementation of API of serial port flow control configuration. ARMmbed#3041 3092: [tools + tests] Adding parallelized build option for iar and uvision exporters ARMmbed#3092 3084: [nrf5] fix in Digital I/O : a gpioe pin was uninitialized badly ARMmbed#3084 3009: TRNG enabled. TRNG APIs implemented. REV A/B/C/D flags removed. Warnings removed ARMmbed#3009 3139: Handle [NOT_SUPPORTED] exception in make.py ARMmbed#3139 3074: Target stm init gcc alignement ARMmbed#3074 3140: [tests] Replacing getchar with RawSerial getc in greentea-client ARMmbed#3140 3158: Added support for 6lowpan PAN ID filter to mbed mesh api configuration ARMmbed#3158 2988: Update of can_api.c fixing ARMmbed#2987 ARMmbed#2988 3175: Updating IAR definition for the NCS36510 for IAR EW v7.8 ARMmbed#3175 3170: [tests] Preventing test from printing before Greentea __sync ARMmbed#3170 3169: [Update of ARMmbed#3014] Usb updates ARMmbed#3169 3143: CFStore fix needed for the Cloud Client ARMmbed#3143 3135: lwip - Fix memory leak in k64f cyclic-buffer overflow ARMmbed#3135 3048: Make update.py test compile examples prior to updating mbed-os version. ARMmbed#3048 3162: lwip/nsapi - Clean up warnings in network code ARMmbed#3162 3161: nsapi - Add better heuristic for the default record of DNS queries ARMmbed#3161 3173: [Exporters] Add a device_name to microbit entry in targets.json ARMmbed#3173 3072: i2c_loop tests update for STM32 ARMmbed#3072 2958: Allowing mbed_app.json files to be discovered for tests. ARMmbed#2958 2969: [nRF52] - switch irq priorities of driver handlers to the lowest level ARMmbed#2969 3078: lwip: Allow several configuration macros to be set externally (bis) ARMmbed#3078 3165: Add address type checks to NanostackInterface ARMmbed#3165 3166: nsapi_dns: Provide 2 IPv6-hosted default servers ARMmbed#3166 3171: [tools] Fixing project.py -S printing problem ARMmbed#3171 3172: [Exporters] New export-build tests ARMmbed#3172 3184: ARMmbed#3183 Compiler warning in trng_api.c with K64F ARMmbed#3184 3185: Update tests to fix build failures. Also make the code similar to oth ARMmbed#3185 3104: [NuMaker] Support CAN and fix PWM CLK error ARMmbed#3104 3182: Exporter documentation ARMmbed#3182 3186: MultiTech mDot - add back SPI3 pins ARMmbed#3186 3187: [Export-Make] Use internal class variable for resolving templates in makefiles ARMmbed#3187 3195: [Exporters - Make-based] Quote the shell call in mkdir and rmdir ARMmbed#3195 3204: [Export build-test] Directory traversal error ARMmbed#3204 3189: [Exporters - Make-based] Force make exporter to search PATH for compilers ARMmbed#3189 3200: Using Popen for uVision and unifying the structure of the build function ARMmbed#3200 3075: nsapi - Add standardized return types for size and errors ARMmbed#3075 3221: u-blox odin w2 drivers update ARMmbed#3221

bridadan reviewed Oct 17, 2016

View reviewed changes

bridadan added needs: review needs: CI labels Oct 17, 2016

Review: Update example_lib.py to use rmtree rather than rm system call.

3696456

Update examples.json to limit TLS examples to a couple of boards and only GCC_ARM and ARM compilers.

theotherjimmy reviewed Oct 21, 2016

View reviewed changes

0xc0170 added needs: work and removed needs: CI needs: review labels Oct 24, 2016

adbridge added needs: review and removed needs: work labels Oct 25, 2016

bridadan approved these changes Oct 25, 2016

View reviewed changes

Update ble example meta data to fix unsupported compilation combinati…

a5d3d20

…ons. Remove URI beacon example as this is no longer required. Add specific set of supported targets.

bridadan added do not merge needs: CI and removed needs: review labels Oct 27, 2016

0xc0170 removed do not merge needs: CI labels Oct 28, 2016

0xc0170 reviewed Oct 28, 2016

View reviewed changes

Remove superfluous debugging line.

c60b19e

bridadan suggested changes Oct 28, 2016

View reviewed changes

Switch off compilation of wifi example while awaiting a fix for

479e203

issue 3152 is submitted.

bridadan approved these changes Oct 28, 2016

View reviewed changes

bridadan added the needs: CI label Oct 28, 2016

0xc0170 added ready for merge and removed needs: CI labels Oct 31, 2016

sg- merged commit fa314e2 into ARMmbed:master Nov 1, 2016

sg- added release-version: 5.2.2 and removed ready for merge labels Nov 1, 2016

sarahmarshy mentioned this pull request Nov 2, 2016

[Exporters] New export-build tests #3172

Merged

3 tasks

Make update.py test compile examples prior to updating mbed-os version. #3048

Make update.py test compile examples prior to updating mbed-os version. #3048

Conversation

adbridge commented Oct 17, 2016

bridadan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bridadan commented Oct 17, 2016

bridadan commented Oct 17, 2016

mbed-bot commented Oct 18, 2016

Output

adbridge commented Oct 18, 2016 • edited Loading

adbridge commented Oct 18, 2016

mbed-bot commented Oct 18, 2016

adbridge commented Oct 18, 2016

theotherjimmy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xc0170 commented Oct 24, 2016

adbridge commented Oct 24, 2016

adbridge commented Oct 25, 2016

bridadan left a comment

Choose a reason for hiding this comment

bridadan commented Oct 25, 2016

mbed-bot commented Oct 25, 2016

Output

bridadan commented Oct 25, 2016

adbridge commented Oct 26, 2016

adbridge commented Oct 26, 2016

mbed-bot commented Oct 26, 2016

Output

bridadan commented Oct 27, 2016

mbed-bot commented Oct 27, 2016

Output

bridadan commented Oct 27, 2016

0xc0170 commented Oct 28, 2016

adbridge commented Oct 28, 2016

mbed-bot commented Oct 28, 2016

Output

adbridge commented Oct 28, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xc0170 commented Oct 28, 2016

bridadan left a comment • edited Loading

Choose a reason for hiding this comment

adbridge commented Oct 28, 2016

adbridge commented Oct 28, 2016

bridadan left a comment • edited Loading

Choose a reason for hiding this comment

mbed-bot commented Oct 28, 2016

Output

sg- commented Nov 1, 2016

adbridge commented Oct 18, 2016 •

edited

Loading

adbridge commented Oct 28, 2016 •

edited

Loading

bridadan left a comment •

edited

Loading

bridadan left a comment •

edited

Loading