New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tools: Extract resources object and implement incremental scan #7183

Merged
merged 44 commits into from Jul 17, 2018

Conversation

Projects
None yet
5 participants
@theotherjimmy
Contributor

theotherjimmy commented Jun 11, 2018

Description

I found that the resource object being part of the tool chain, and depending on
tool chain infrastructure was a bit odd, so I extracted resources from tool
chains. The resulting code is quite a bit more clear in it's purpose and can be a
tad smaller (extra documentation and formatting excluded). I made the resources
object fully incremental, meaning that the scan stops early otfen for targets,
toolchains and features, and can easily pick itself back up where it left off.

A side effect of fully incremental scan is that configuration, which cannot depend
on a tool chain, does not need a tool chain specified to work correctly. I updated
get_config.py (mbed compile --config's back end) to reflect this.

Fixes #6038 (--source with a path starting with ../ shows incorrectly in memap)
Fixes #5357 (--source with uvision exporter creates broken project files)
(Possibly more as I find issues fixed by being up front about our data model)

TODO

  • replace that odd file_basepath member with something more suitable
  • finish replacing calls to toolchain.scan_resources and scan_resources
  • Document new architecture.

Pull request type

[ ] Fix
[x] Refactor
[ ] New target
[ ] Feature
[ ] Breaking change

@0xc0170 0xc0170 requested a review from ARMmbed/mbed-os-tools Jun 12, 2018

@theotherjimmy theotherjimmy force-pushed the theotherjimmy:refactor-resources branch 2 times, most recently from b52881d to 1fe20b1 Jun 15, 2018

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jun 18, 2018

@cmonr 🤞 I got the last "use paths" bug. That should unblock Oulu CI. Ready for review.

@cmonr

LGTM. A couple of nits and questions, but should be good to go afterwards.

@@ -0,0 +1 @@
jimmy@archlinux.3808:1528302063

This comment has been minimized.

@cmonr

cmonr Jun 18, 2018

Contributor

Something tells me this shouldn't be here

This comment has been minimized.

@theotherjimmy

theotherjimmy Jun 19, 2018

Contributor

Oh for sure. I'm going to rebase and purge it from the history.

# library
# A number of compiled files need to be copied as objects as opposed to
# way the linker search for symbols in archives. These are:

This comment has been minimized.

@cmonr

cmonr Jun 18, 2018

Contributor
  • the way

This comment has been minimized.

@theotherjimmy

theotherjimmy Jun 19, 2018

Contributor

lol. I'll fix that.

@@ -976,141 +907,138 @@ def build_mbed_libs(target, toolchain_name, clean=False, macros=None,
properties - UUUUHHHHH beats me

This comment has been minimized.

@cmonr

cmonr Jun 18, 2018

Contributor

This should probably get a better description at some point.

This comment has been minimized.

@theotherjimmy

theotherjimmy Jun 19, 2018

Contributor

At some point, when I see it used. (I have not seen it passed in)

This comment has been minimized.

@theotherjimmy

theotherjimmy Jun 19, 2018

Contributor

I just looked around, and it seems that there were plans for a thing called properties, but no implementation using them...

mbed_resources = Resources(notify).scan_with_toolchain(
[MBED_DRIVERS, MBED_PLATFORM, MBED_HAL], toolchain)
incdirs = cmsis_res.inc_dirs + hal_res.inc_dirs + library_incdirs

This comment has been minimized.

@cmonr

cmonr Jun 18, 2018

Contributor

Beautiful

This comment has been minimized.

@theotherjimmy

theotherjimmy Jun 19, 2018

Contributor

Thanks.

@theotherjimmy theotherjimmy force-pushed the theotherjimmy:refactor-resources branch 2 times, most recently from b4a0ba2 to ba35a00 Jun 19, 2018

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jun 19, 2018

@cmonr I rebased out those problems that you commented on.

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jun 19, 2018

Master:

$ wc ./BUILD/K64F/ARM/.includes_2e382264abc50dd02247baa7de4a1173.txt
    0   241 15017 ./BUILD/K64F/ARM/.includes_2e382264abc50dd02247baa7de4a1173.txt

This branch:

$ wc ./BUILD/K64F/ARM/.includes_05d475aa5eadf631d0b7eb0ab4459616.txt
    0   195 11711 ./BUILD/K64F/ARM/.includes_05d475aa5eadf631d0b7eb0ab4459616.txt

@theotherjimmy theotherjimmy force-pushed the theotherjimmy:refactor-resources branch 2 times, most recently from 2a0e535 to 0942442 Jun 19, 2018

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jun 19, 2018

mbed test not working. impl of copy_sources used to use file_basepath, now it needs an update.

@theotherjimmy theotherjimmy force-pushed the theotherjimmy:refactor-resources branch from d1d720c to 32398d4 Jun 19, 2018

@0xc0170 0xc0170 added needs: work and removed needs: review labels Jun 20, 2018

@cmonr cmonr dismissed their stale review Jun 21, 2018

Some changes have been made since last review.

@theotherjimmy theotherjimmy force-pushed the theotherjimmy:refactor-resources branch from 32398d4 to fa2369c Jun 22, 2018

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jun 22, 2018

rebased.

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jun 25, 2018

@cmonr Ready for review.

if options.tool is None:
args_error(parser, "argument -t/--toolchain is required")
toolchain = options.tool[0]
toolchain = options.tool[0] if options.tool is not None else None

This comment has been minimized.

@cmonr

cmonr Jun 26, 2018

Contributor

Any reason this couldn't be shrunk to options.tool[0] if options.tool else None?

This comment has been minimized.

@theotherjimmy

theotherjimmy Jun 26, 2018

Contributor

No. Originally we had to worry about empty strings and what not.

(headername, " ".join(locations)))
return count
def win_to_unix(self):

This comment has been minimized.

@cmonr

cmonr Jun 26, 2018

Contributor

💯

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jul 16, 2018

All conflicts were with my other PRs...

@cmonr

This comment has been minimized.

Contributor

cmonr commented Jul 16, 2018

/morph build

@cmonr cmonr added needs: CI and removed needs: review labels Jul 16, 2018

@mbed-ci

This comment has been minimized.

mbed-ci commented Jul 16, 2018

Build : SUCCESS

Build number : 2619
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7183/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

@mbed-ci

This comment has been minimized.

@mbed-ci

This comment has been minimized.

@cmonr cmonr merged commit c29fe89 into ARMmbed:master Jul 17, 2018

14 checks passed

AWS-CI uVisor Build & Test Success
Details
ci-morph-build build completed
Details
ci-morph-exporter build completed
Details
ci-morph-mbed2-build build completed
Details
ci-morph-test test completed
Details
continuous-integration/jenkins/pr-head This commit looks good
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
travis-ci/astyle Passed, 793 files
Details
travis-ci/docs Local docs testing has passed
Details
travis-ci/events Passed, runtime is 10213 cycles (+202 cycles)
Details
travis-ci/gitattributestest Local gitattributestest testing has passed
Details
travis-ci/licence_check Local licence_check testing has passed
Details
travis-ci/littlefs Passed, code size is 9960B (+0.00%)
Details
travis-ci/tools-py2.7 Local tools-py2.7 testing has passed
Details

@0xc0170 0xc0170 referenced this pull request Jul 20, 2018

Closed

mbed-cli include issue #7560

@cmonr

This comment has been minimized.

Contributor

cmonr commented Jul 27, 2018

Looks like this PR has a dependency on another PR targeted for 5.10.

@theotherjimmy

This comment has been minimized.

Contributor

theotherjimmy commented Jul 27, 2018

As usual, dependency might be the wrong word here. It was definitely rebased on top of mayn other PRs.

@cmonr

This comment has been minimized.

Contributor

cmonr commented Jul 28, 2018

Correct, it was. Not sure what a better term would be, but what I do know was that this PR made changes on top of this (#7133) PR, which caused pytest and build failures.

pan- pushed a commit to pan-/mbed that referenced this pull request Aug 22, 2018

Merge pull request ARMmbed#7183 from theotherjimmy/refactor-resources
Tools: Extract resources object and implement incremental scan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment