Tools: Extract resources object and implement incremental scan #7183

theotherjimmy · 2018-06-11T16:35:04Z

Description

I found that the resource object being part of the tool chain, and depending on
tool chain infrastructure was a bit odd, so I extracted resources from tool
chains. The resulting code is quite a bit more clear in it's purpose and can be a
tad smaller (extra documentation and formatting excluded). I made the resources
object fully incremental, meaning that the scan stops early otfen for targets,
toolchains and features, and can easily pick itself back up where it left off.

A side effect of fully incremental scan is that configuration, which cannot depend
on a tool chain, does not need a tool chain specified to work correctly. I updated
get_config.py (mbed compile --config's back end) to reflect this.

Fixes #6038 (--source with a path starting with ../ shows incorrectly in memap)
Fixes #5357 (--source with uvision exporter creates broken project files)
(Possibly more as I find issues fixed by being up front about our data model)

TODO

replace that odd file_basepath member with something more suitable
finish replacing calls to toolchain.scan_resources and scan_resources
Document new architecture.

Pull request type

[ ] Fix
[x] Refactor
[ ] New target
[ ] Feature
[ ] Breaking change

theotherjimmy · 2018-06-18T16:44:01Z

@cmonr 🤞 I got the last "use paths" bug. That should unblock Oulu CI. Ready for review.

cmonr

LGTM. A couple of nits and questions, but should be good to go afterwards.

cmonr · 2018-06-18T17:09:05Z

tools/.#make.py

@@ -0,0 +1 @@
+jimmy@archlinux.3808:1528302063


Something tells me this shouldn't be here

Oh for sure. I'm going to rebase and purge it from the history.

cmonr · 2018-06-18T18:29:06Z

tools/build_api.py

-# library
+
+# A number of compiled files need to be copied as objects as opposed to
+# way the linker search for symbols in archives. These are:


lol. I'll fix that.

cmonr · 2018-06-18T18:33:32Z

tools/build_api.py

@@ -976,141 +907,138 @@ def build_mbed_libs(target, toolchain_name, clean=False, macros=None,
    properties - UUUUHHHHH beats me


This should probably get a better description at some point.

At some point, when I see it used. (I have not seen it passed in)

I just looked around, and it seems that there were plans for a thing called properties, but no implementation using them...

cmonr · 2018-06-18T18:38:51Z

tools/build_api.py

+        mbed_resources = Resources(notify).scan_with_toolchain(
+            [MBED_DRIVERS, MBED_PLATFORM, MBED_HAL], toolchain)
+
+        incdirs = cmsis_res.inc_dirs + hal_res.inc_dirs + library_incdirs


theotherjimmy · 2018-06-19T14:57:55Z

@cmonr I rebased out those problems that you commented on.

theotherjimmy · 2018-06-19T18:52:37Z

Master:

$ wc ./BUILD/K64F/ARM/.includes_2e382264abc50dd02247baa7de4a1173.txt
    0   241 15017 ./BUILD/K64F/ARM/.includes_2e382264abc50dd02247baa7de4a1173.txt

This branch:

$ wc ./BUILD/K64F/ARM/.includes_05d475aa5eadf631d0b7eb0ab4459616.txt
    0   195 11711 ./BUILD/K64F/ARM/.includes_05d475aa5eadf631d0b7eb0ab4459616.txt

theotherjimmy · 2018-06-19T19:41:05Z

mbed test not working. impl of copy_sources used to use file_basepath, now it needs an update.

Some changes have been made since last review.

theotherjimmy · 2018-06-22T15:22:28Z

rebased.

theotherjimmy · 2018-06-25T15:35:24Z

@cmonr Ready for review.

cmonr · 2018-06-26T13:21:48Z

tools/get_config.py

-    if options.tool is None:
-        args_error(parser, "argument -t/--toolchain is required")
-    toolchain = options.tool[0]
+    toolchain = options.tool[0] if options.tool is not None else None


Any reason this couldn't be shrunk to options.tool[0] if options.tool else None?

No. Originally we had to worry about empty strings and what not.

cmonr · 2018-06-26T13:28:36Z

tools/resources/__init__.py

+                    (headername, " ".join(locations)))
+        return count
+
+    def win_to_unix(self):


theotherjimmy · 2018-07-16T19:12:37Z

All conflicts were with my other PRs...

cmonr · 2018-07-16T19:12:58Z

/morph build

mbed-ci · 2018-07-16T20:51:04Z

Build : SUCCESS

Build number : 2619
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7183/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-07-16T21:55:53Z

Exporter Build : SUCCESS

Build number : 2259
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7183/

mbed-ci · 2018-07-17T02:00:33Z

Test : SUCCESS

Build number : 2368
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7183/2368

cmonr · 2018-07-27T16:49:15Z

Looks like this PR has a dependency on another PR targeted for 5.10.

theotherjimmy · 2018-07-27T20:30:24Z

As usual, dependency might be the wrong word here. It was definitely rebased on top of mayn other PRs.

cmonr · 2018-07-28T01:14:11Z

Correct, it was. Not sure what a better term would be, but what I do know was that this PR made changes on top of this (#7133) PR, which caused pytest and build failures.

Tools: Extract resources object and implement incremental scan

theotherjimmy added needs: work component: tools labels Jun 11, 2018

0xc0170 requested a review from a team June 12, 2018 08:10

theotherjimmy force-pushed the refactor-resources branch 2 times, most recently from b52881d to 1fe20b1 Compare June 15, 2018 18:43

theotherjimmy added needs: review and removed needs: work labels Jun 18, 2018

cmonr previously requested changes Jun 18, 2018

View reviewed changes

theotherjimmy force-pushed the refactor-resources branch 2 times, most recently from b4a0ba2 to ba35a00 Compare June 19, 2018 14:57

theotherjimmy mentioned this pull request Jun 19, 2018

Upgrade memap to show components of included libraries #6038

Closed

theotherjimmy force-pushed the refactor-resources branch 2 times, most recently from 2a0e535 to 0942442 Compare June 19, 2018 19:29

theotherjimmy force-pushed the refactor-resources branch from d1d720c to 32398d4 Compare June 19, 2018 20:26

0xc0170 added needs: work and removed needs: review labels Jun 20, 2018

theotherjimmy mentioned this pull request Jun 20, 2018

Uvision5: export from within a sub folder #5357

Closed

theotherjimmy force-pushed the refactor-resources branch from 32398d4 to fa2369c Compare June 22, 2018 15:22

theotherjimmy added needs: review and removed needs: work labels Jun 25, 2018

cmonr reviewed Jun 26, 2018

View reviewed changes

tools/resources/__init__.py

(headername, " ".join(locations)))

return count

def win_to_unix(self):

Copy link

Contributor

cmonr Jun 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

cmonr previously approved these changes Jun 26, 2018

View reviewed changes

theotherjimmy force-pushed the refactor-resources branch from 8af900c to 750a2ac Compare July 16, 2018 19:12

theotherjimmy added needs: review and removed needs: work labels Jul 16, 2018

cmonr added needs: CI and removed needs: review labels Jul 16, 2018

cmonr added ready for merge release-version: 5.10.0-rc1 release-version: 5.9.4 and removed needs: CI release-version: 5.9.4 release-version: 5.10.0-rc1 labels Jul 17, 2018

cmonr merged commit c29fe89 into ARMmbed:master Jul 17, 2018

0xc0170 removed the ready for merge label Jul 17, 2018

0xc0170 mentioned this pull request Jul 20, 2018

mbed-cli include issue #7560

Closed

This was referenced Jul 25, 2018

Export: Impl Makefile-based response/command files #7583

Closed

Windows makefile builds no longer work #7594

Closed

theotherjimmy mentioned this pull request Jul 26, 2018

Tools: Correct win_to_unix path handling #7608

Merged

cmonr added release-version: 5.10.0-rc1 and removed release-version: 5.9.4 labels Jul 27, 2018

adbridge mentioned this pull request Aug 10, 2018

Tools: Omit include path parents when they're scan rules #7623

Merged

jeromecoutant mentioned this pull request Aug 10, 2018

OS2 tests scripts doesn't work any more #7756

Closed

pan- pushed a commit to pan-/mbed that referenced this pull request Aug 22, 2018

Merge pull request ARMmbed#7183 from theotherjimmy/refactor-resources

d046bd8

Tools: Extract resources object and implement incremental scan

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tools: Extract resources object and implement incremental scan #7183

Tools: Extract resources object and implement incremental scan #7183

theotherjimmy commented Jun 11, 2018 •

edited

Loading

theotherjimmy commented Jun 18, 2018

cmonr left a comment

cmonr Jun 18, 2018

theotherjimmy Jun 19, 2018

cmonr Jun 18, 2018

theotherjimmy Jun 19, 2018

cmonr Jun 18, 2018

theotherjimmy Jun 19, 2018

theotherjimmy Jun 19, 2018

cmonr Jun 18, 2018

theotherjimmy Jun 19, 2018

theotherjimmy commented Jun 19, 2018

theotherjimmy commented Jun 19, 2018

theotherjimmy commented Jun 19, 2018

theotherjimmy commented Jun 22, 2018

theotherjimmy commented Jun 25, 2018

cmonr Jun 26, 2018

theotherjimmy Jun 26, 2018

cmonr Jun 26, 2018

theotherjimmy commented Jul 16, 2018

cmonr commented Jul 16, 2018

mbed-ci commented Jul 16, 2018

mbed-ci commented Jul 16, 2018

mbed-ci commented Jul 17, 2018

cmonr commented Jul 27, 2018

theotherjimmy commented Jul 27, 2018

cmonr commented Jul 28, 2018

		@@ -976,141 +907,138 @@ def build_mbed_libs(target, toolchain_name, clean=False, macros=None,
		properties - UUUUHHHHH beats me

		@@ -0,0 +1 @@
		jimmy@archlinux.3808:1528302063

Tools: Extract resources object and implement incremental scan #7183

Tools: Extract resources object and implement incremental scan #7183

Conversation

theotherjimmy commented Jun 11, 2018 • edited Loading

Description

TODO

Pull request type

theotherjimmy commented Jun 18, 2018

cmonr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theotherjimmy commented Jun 19, 2018

theotherjimmy commented Jun 19, 2018

theotherjimmy commented Jun 19, 2018

theotherjimmy commented Jun 22, 2018

theotherjimmy commented Jun 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theotherjimmy commented Jul 16, 2018

cmonr commented Jul 16, 2018

mbed-ci commented Jul 16, 2018

Triggering tests

mbed-ci commented Jul 16, 2018

mbed-ci commented Jul 17, 2018

cmonr commented Jul 27, 2018

theotherjimmy commented Jul 27, 2018

cmonr commented Jul 28, 2018

theotherjimmy commented Jun 11, 2018 •

edited

Loading