Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FUTURE_SEQUANA: failing make nightly exporters #9611

Open
0xc0170 opened this Issue Feb 5, 2019 · 23 comments

Comments

Projects
None yet
6 participants
@0xc0170
Copy link
Member

0xc0170 commented Feb 5, 2019

Description

Logs here http://mbed-os-ci.s3-website-eu-west-1.amazonaws.com/?prefix=jenkins-ci/artifacts/master/107/exporter/FAIL/

Check make_ exporters. srcat is failing in the last step.

One of these failures:

srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500000 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500001 value
    (0x05)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500002 value
    (0xE2)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500003 value
    (0x07)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500004 value
    (0x21)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500005 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500006 value
    (0x21)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500007 value
    (0x01)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500008 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: warning: redundant 0x90500009 value
    (0x00)
srec_cat: mbed-os-example-blinky.hex: 3850: contradictory 0x9050000A value
    (previous = 0x45, this one = 0x00)
make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2

or

Finished: 0 information, 9 warning and 0 error messages.
srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory

@orenc17 @alzix Who could help fixing these?

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug
@orenc17

This comment has been minimized.

Copy link
Contributor

orenc17 commented Feb 5, 2019

@0xc0170 i believe @lrusinowicz could help

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Feb 5, 2019

This is affecting our nightly, it would be good to have it fixed soon

@lrusinowicz

This comment has been minimized.

Copy link
Contributor

lrusinowicz commented Feb 5, 2019

@0xc0170
Martin, is this with build on the master/HEAD?
I believe I have a potential fix, but how can I test it helps?

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Feb 5, 2019

Correct, master HEAD is being tested. you should be able to access the logs. Use mbed-os-example-blinky and export for failing exporters, to reproduce the errors

@ciarmcom ciarmcom added the type: bug label Feb 5, 2019

@ciarmcom

This comment has been minimized.

Copy link
Member

ciarmcom commented Feb 5, 2019

lrusinowicz added a commit to lrusinowicz/mbed-os that referenced this issue Feb 5, 2019

adbridge added a commit that referenced this issue Feb 8, 2019

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Feb 11, 2019

Fixed via referenced PR

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Feb 19, 2019

@lrusinowicz I've fetched latest nightly results and we can still seeing this failing for ARMC5 on master

Finished: 0 information, 9 warning and 0 error messages.
srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory
make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
make: *** [all] Error 2

This is the failing example Exporting mbed-os-example-blinky FUTURE_SEQUANA make_armc5. Can you please review and reproduce?

@0xc0170 0xc0170 reopened this Feb 19, 2019

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Feb 19, 2019

@lrusinowicz

This comment has been minimized.

Copy link
Contributor

lrusinowicz commented Feb 25, 2019

@0xc0170
The log content is not giving any useful hints on why this might be failing.
Does this export test succeed on any other PSOC6 target, i.e. is a failure specific to FUTURE_SEQUANA target?
What is the environment (operating system, make version) used on your cloud servers?
I actually can't reproduce the exact issue at the moment. I don't have access to ARM compiler on my Linux system and the exported makefile doesn't work on Windows 7 with GNU make versions 4.1/4.2 because of different issue (command line being evidently truncated on generation of .link_options.txt

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Feb 25, 2019

What is the environment (operating system, make version) used on your cloud servers?

The job run on linux node. I cant find make being printed in the logs.
The command python -u ../mbed-os/tools/test/examples/examples.py export make_armc5 --mcu FUTURE_SEQUANA_PSA .

From the log what I can read is that srec_cat fails, linking is fine

03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] Finished: 0 information, 9 warning and 0 error messages.
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] make: *** [all] Error 2
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] FAILURE
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] FAILURE building mbed-os-example-blinky FUTURE_SEQUANA_PSA make_armc5
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] # Examples compilation summary
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] #
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] # Failed build combinations
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] # mbed-os-example-blinky FUTURE_SEQUANA_PSA make_armc5
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] #
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] ################################################################################
03:25:20 03:25:20 [FUTURE_SEQUANA_PSA:make_armc5] Number of failures = 1

Does this export test succeed on any other PSOC6 target, i.e. is a failure specific to FUTURE_SEQUANA target?

Only FUTURE_SEQUANA and FUTURE_SEQUANA_PSA are affected.

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Feb 25, 2019

@ARMmbed/mbed-os-test Need your help to get more details for this nightly failures

@cmonr

This comment has been minimized.

Copy link
Contributor

cmonr commented Feb 25, 2019

@cmonr

This comment has been minimized.

Copy link
Contributor

cmonr commented Feb 25, 2019

Finished: 0 information, 9 warning and 0 error messages.
srec_cat: mbed-os-example-blinky.hex: 1: read: Is a directory
make[1]: *** [mbed-os-example-blinky-combined.hex] Error 1
make: *** [all] Error 2
FAILURE
FAILURE building mbed-os-example-blinky FUTURE_SEQUANA make_armc5

I recall a specific version of srec_cat being needed. @lrusinowicz does that sound accurate?
@studavekar Does that sound famililar?

@ARMmbed/mbed-os-test Fyi

All: Something that appears related: #8491 (comment)

@studavekar

This comment has been minimized.

Copy link
Contributor

studavekar commented Feb 25, 2019

I recall a specific version of srec_cat being needed. @lrusinowicz does that sound accurate?
@studavekar Does that sound famililar?

Yeah, NRF devices are passing nightly even they use scre_cat. maybe its something else.

@cmonr

This comment has been minimized.

Copy link
Contributor

cmonr commented Feb 25, 2019

@ARMmbed/mbed-os-test Are we able to tell what versions of python modules are in use in nightly?

@cmonr

This comment has been minimized.

Copy link
Contributor

cmonr commented Feb 25, 2019

Whoops, looks like this is a native shell utility.

@ARMmbed/mbed-os-test-team Any way we can run srec_cat --version in the nightly VM?

@lrusinowicz

This comment has been minimized.

Copy link
Contributor

lrusinowicz commented Feb 26, 2019

Do you also test exporter for make_gcc_arm? If yes, then does it work?

I was testing make_gcc_arm exporter on Linux locally (don't have ARMCC on Linux) and the resulting makefile is working correctly, including building a combined image with srec_cat.
The combined image section (and resulting srec_cat command) seems to be identical for both, GCC and ARMCC makefiles.
Yes, I also recall we had some issue with srec_cat version before, but I cannot find any details.
I have srec_cat version 1.58.D001 and GNU make 3.81 on my Linux system.

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Mar 7, 2019

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Mar 7, 2019

@ARMmbed/mbed-os-test-team Any way we can run srec_cat --version in the nightly VM?

@alekla01 @timurh01 Please help us why is srec_cat failing in the CI.

This looks like specific to FUTURE_SEQUANA boards as other targets that use srec_cat are working.

@lrusinowicz

This comment has been minimized.

Copy link
Contributor

lrusinowicz commented Mar 7, 2019

@0xc0170
I see two, separate issues in the build logs:
1. gcc/FUTURE_SEQUANA_PSA
srec_cat fails, because re-generated PSA M0 images contain special addresses starting with 0x9******, conflicting with the same addresses generated for M4 image. These can be just manually cut-out from M0 hex files, but I will submit a change, that should stop generation of these addresses in the first place.
2. armc5/FUTURE_SEQUANA (original issue)
This one is failing, because somehow BUILD directory structure is non-consistent with makefile srec_cat command. I will need to do more analysis here. But this is somewhat difficult, as I do not have possibility to reproduce this on Linux and on Windows this fails for different reasons (too long paths, if I remember correctly).

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Mar 7, 2019

Thanks for the update and looking at these logs. The fixes should target 5.12rc2 (reference this issue there so we are aware of the proper release target).

rajkan01 added a commit to rajkan01/mbed-os that referenced this issue Mar 7, 2019

@lrusinowicz

This comment has been minimized.

Copy link
Contributor

lrusinowicz commented Mar 8, 2019

@0xc0170
Strange build failure in armc5 builds was caused by specific behaviour of ARM fromelf utility and the option used in exported makefile --i32. With this option, when sparse sections are present in binary, the tool generates multiple hex files within the directory named according to the output option, hence srec_cat was complaining that it was given directory instead of a file name.
Python build scripts use different option --i32combined to prevent this from happening.
Anyway, with cymetadata section removed with #10011, sparse sections will not be present and either option will work. Removal of metadata should also resolve FUTURE_SEQUANA_PSA failures even without re-build of PSA binaries, because there will be no conflicting data in the main program.

@0xc0170

This comment has been minimized.

Copy link
Member Author

0xc0170 commented Mar 8, 2019

Thanks for fixing this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.