allow sharing static libraries (esp. mbed-os) among multiple projects

[bmcdonnell-ionx]

This is a feature(s) request, originally discussed here: How can I factor mbed-os out of my (offline) project?.

Currently, AFAIK, there is no way to use a single copy of a static library from multiple projects. I think it would be quite useful to be able to do this both entirely within mbed-cli, and also in exported projects (using offline IDEs). I think it would be especially useful to have mbed-os as a static library or libraries, which are also buildable as an exported static library project(s).

Suppose I have executable ProjectBar, which links to static library LibFoo (which could be mbed-os-lib). I'm requesting these features:

1. Ability to link to LibFoo within ProjectBar, while LibFoo resides outside the ProjectBar sourcetree. (i.e. Link to the already-built library and header files.)
   • mbed-cli compile links in LibFoo.
   • Can use relative and/or absolute paths.
2. Ability to export LibFoo to offline IDEs as a static library project.
3. When exporting ProjectBar to an offline IDE, maintain references to LibFoo in the project/linker files.
4. Ability to "step into" LibFoo source when debugging ProjectBar.

Of course, the last one is lowest priority relative to the others - kind of a "nice to have".

[bmcdonnell-ionx]

Related: #547.

[janjongboom]

@bmcdonnell-ionx So, one way would be to build the static library from a location, then link the static binary in through a symbolic link. Just a bit of work in initial setup, but after that it does what you want to do right?

[bmcdonnell-ionx]

@janjongboom IT is heavily restricting local admin privileges on our PCs, so that would not be a practical solution for us.

https://github.com/git-for-windows/git/wiki/Symbolic-Links

Also, it would be preferable to have a group of repos that users can just clone and batch build. Clever, though.

[bmcdonnell-ionx]

Let me draw a parallel here: consider MS Visual Studio solutions and projects.

Solutions (the example)

Each project produces one output, be it an executable or a library. It is good practice to make projects modular. The solutions track dependencies between projects.

You could have a setup like this:

$ ls -F MySuites/
App1/
App2/
App3/
libBar/
libFoo/
SuiteAll.sln
SuiteMinimal.sln

MySuites is the top-level folder containing related solutions and projects. App[123] are projects that produce executables - each in their own folder. lib{Foo,Bar} are projects that produce libraries (say static libraries). The SuiteAll.sln solution builds all 3 Apps; SuiteMinimal.sln builds just App1.

App1 uses libFoo; App2 uses libBar; App3 uses both. etc.

Each module (executable or library project) only gets built once, and modules can reference other modules which need not be in subfolders beneath the projects that reference them.

Mbed Land (the idea/proposal)

I think it would be great if mbed-cli could do this sort of thing - both from mbed compile, and also mbed export. For instance, to imitate the example above, maybe you'd have a top-level repo like this:

$ ls -aF MySuites/
.
..
.git/
app1.lib
app2.lib
app3.lib
lib-bar.lib
lib-foo.lib
mbed-os.lib
suite-all.json
suite-minimal.json

Like it does now, mbed import would clone the repos referenced by all the .lib files. (Note, there's an overloading of the term "library" here - meaning either [1] a static library project (e.g. lib-bar), or [2] an Mbed .lib file that points to a repo (lib-bar.lib or app1.lib). Perhaps you'd come up with a different file extension.)

Anyway, passing one (or more?) of these .json files to mbed compile would build all the projects referenced in file(s), and link them according to their dependencies. (Perhaps mbed build would be a better name.) Likewise, passing one of these .json file(s) to mbed export would create the multiple projects, and whatever metadata tracks their dependencies (analog of the "solution") in the target IDE. That way you don't have multiple copies of these projects (mbed-os!) consuming disk space, and build time. AND you can still debug in the application and library code.

It sounds like a non-trivial effort, but I think it would be a leap and bound in terms of development scalability.

Feedback

Mbed OS team: what do you think? Do you consider the current setup "good enough", where the whole mbed-os source tree gets replicated into every project? Do you have interest or plans to make this more flexible, or even do something like I've described here?

[bridadan]

@bmcdonnell-ionx This almost works as you described as long as libBar and libFoo are git repositories.

All you have to do is the following:

cd App1
mbed add /absolute/path/to/libFoo

That will clone libFoo into the current App1. The downside is this:

1. requires you to use an absolute path, so it will only work on the current machine (as opposed to using remote repositories, where they are cloned over the network).
2. currently incorrectly saves the path with https:// (ex. on my Windows machine the saved .lib file looks like https://c//Users/bridan01/onedrive_arm/Documents/dev/lib_test/mylib/#e8ae428ee4d63433ac56c6218f557e0c458775db)

If we were to solve at least #2, Would that be a workable solution to your issue?

[bridadan]

Just as an update to my previous post, good news! #2 is actually already fixed on master!

It's important to note, however, that if you make an update to a shared library in App1, this update won't be reflected in App2 since that is a separate clone of the library repository. If you want it to be updated in App2, you should commit and push your changes to the shared library and then update the library inside App2 by pulling down these changes.

The upside to this approach is that you can avoid breaking App2 and App3 and still experiment with changes inside the shared lib within App1. The downside is that you need to do a few more steps to keep all of these libraries in sync.

The build tools will follow symbolic links I believe, so if you really want the source to be shared (and not have to do all the git commits) then you can do that. But just realize there is a bit more risk since there is less isolation.

[bmcdonnell-ionx]

@bridadan, thanks for your input.

This almost works as you described as long as libBar and libFoo are git repositories.

By "this", do you mean for single projects as described in my OP? Not a unified multi-project setup as described in my later comment (a master top-level repo containing libs and "solutions"). Assuming so...

mbed add /absolute/path/to/libFoo

It doesn't seem to work, though. With mbed-cli 1.3.0, I tried copying an already-existing project to a new location (test-proj/). I deleted mbed-os.lib from it, moved the mbed-os/ folder up to be a peer with test-proj/. When I executed your suggested mbed add command with -vv, I get

[mbed] ERROR: Directory "C:\dev_temp\test01\test-proj\mbed-os" is not empty. Please ensure that the destination folder is empty.

Would that be a workable solution to your issue?

If it works, it would certainly be a step forward. But is it intentional that it work, or just sort of "accidental"? I've not seen this documented anywhere as a feature, so I'd worry about it breaking. Also, relative paths (#1) would be preferred. How much work would it be to get that working?

However, I think I'd still much prefer my later-proposed more comprehensive idea ("solutions", etc.). I think this would help to keep things in sync, addressing issues like you posed in your "update" comment (immediately preceding this comment of mine). What do you think of it? Can you answer my questions posed there ("feedback" section)?

[theotherjimmy]

@bmcdonnell-ionx Why not just use a symlink? It seems to meet all of your requirements.

[theotherjimmy]

Also, note that object files and static libraries are treated slightly differently WRT weak references by the linker, so we may not be able to package everything as a static library.

[bmcdonnell-ionx]

Why not just use a symlink?

Answered above:

IT is heavily restricting local admin privileges on our PCs, so that would not be a practical solution for us.

https://github.com/git-for-windows/git/wiki/Symbolic-Links

Also, it would be preferable to have a group of repos that users can just clone and batch build. Clever, though.

Details: The Windows 10 Creators Update allows users to create symbolic links on Windows 10 without admin priviliges. Our IT department hasn't pushed it out yet. Alternatively, prior to the update, there is a way for domain admins to grant permission for certain people to use mklink without admin privs, but then using links further reduces portability, and complicates use for those without the capability. Also, some filesystems (e.g. FAT) don't support symbolic links, so you couldn't e.g. have such a working copy on a thumb drive.

Also, if it works as expected, using symlinks would present multiple copies of source files in an IDE, though they'd all be using the same location on disk. Not a huge deal, but sub-optimal, IMO, as the IDE presentation wouldn't reflect the project/library modularity. And there would be no way to build just a library in the IDE.

[bmcdonnell-ionx]

@theotherjimmy,

Also, note that object files and static libraries are treated slightly differently WRT weak references by the linker, so we may not be able to package everything as a static library.

Could you elaborate? How are they treated differently? What problems would it cause? etc.

[bmcdonnell-ionx]

Addendum: using symlinks would also make the development environments more platform dependent, or requires some git hackery to make it play nice when checked out by multiple OSes.

https://stackoverflow.com/questions/5917249/git-symlinks-in-windows

[theotherjimmy]

Could you elaborate? How are they treated differently? What problems would it cause? etc.

Looking at the ARM linker manual, one sentence sticks out at me:

The linker does not load an object from a library to resolve a weak reference.

I'm not clear on how this would resolve weak references within libraries, or if it could even resolve weak references within libraries.

---

Symlinks appear to be supported on Windows, stating with Vista

[theotherjimmy]

there is no way to use a single copy of a static library from multiple projects.

This is by design: the configuration system allows a project to change the contents of the library, which would prevent sharing or make sharing only possible with exactly the same configuration.