Introduce static_to_shared_library function #6092

michaelkuhn · 2017-11-02T00:38:00Z

This is work in progress based on the discussion in #6008.

@junghans: Sorry for the timing. I was just working on this when you merged #6008. :-)

The static_to_shared_library function takes an existing static library and produces a shared library based on it.

For now, I just put it into the lua package. I would like to get some feedback whether the function's interface etc. are what you had in mind. Additionally, I would need some hints on how to integrate this better into Spack. (I have already experimented with putting it into build_environment.py but I currently need access to spack_cc and dso_suffix.)

junghans · 2017-11-02T00:41:25Z

@tgamblin I would really like to see static_to_shared_library as a part of the core, where would we put this?

michaelkuhn · 2017-11-02T01:41:58Z

Small update: I moved it into the package class, which has the benefit of giving us access to the spec.

junghans · 2017-11-02T15:37:24Z

@tgamblin @scheibelp @adamjstewart @alalazo

tgamblin

Ok, several questions/thoughts:

How well does this work across different compilers? Are the arguments portable?
Does this require that the static lib already be PIC? Or does it work in general?
This seems like something that should only exist in the build environment (though if there are other uses maybe we should rethink that). Maybe the right place for it is in build_environment.py, and it could be a function added to the module scope in, e.g., set_module_variables_for_package(). You have the Package and spack_cc there, so you could construct a converter function specialized for the build.
Needs a test.

michaelkuhn · 2017-11-02T20:04:22Z

I have refactored the code a bit, put it into build_environment.py and added a rough first test for it (see below).

How well does this work across different compilers? Are the arguments portable?

From what I could find, at least GCC, Intel and Clang should support -Wl. GCC and Clang support -shared, -dynamiclib and -install_name. The rest depends on the linker.

Does this require that the static lib already be PIC? Or does it work in general?

The static library has to be compiled with PIC.

This seems like something that should only exist in the build environment (though if there are other uses maybe we should rethink that). Maybe the right place for it is in build_environment.py, and it could be a function added to the module scope in, e.g., set_module_variables_for_package(). You have the Package and spack_cc there, so you could construct a converter function specialized for the build.

Done. I moved the compiler selection out of the function to be able to test it more easily.

Needs a test.

I added a rough first test for it that can be extended later. Please check whether this is what you had in mind.

mathstuf · 2017-11-02T20:42:26Z

Open questions:

What to do for Windows? Symbols need to be exported there, so a .def file listing the symbols to export from the shared library is required. Headers may also need patched to set the proper dllimport settings (static libraries tend to have no declspec at all). Blindly listing the symbols for a .def file may end up exporting supposed-to-be-internal symbols as well.
How to handle multiple static libraries in a package? What if they have circular dependencies?
Static builds may have compiled out some features necessary in a shared build (though it seems this is more for coercing never-shared projects to be shared rather than replacing an intentional link=shared option).
Are soname numbers just skipped? What about library IDs on macOS? I see something is set; is that a full path or just the filename?
Other compiler support?

michaelkuhn · 2017-11-02T21:03:49Z

What to do for Windows? Symbols need to be exported there, so a .def file listing the symbols to export from the shared library is required. Headers may also need patched to set the proper dllimport settings (static libraries tend to have no declspec at all). Blindly listing the symbols for a .def file may end up exporting supposed-to-be-internal symbols as well.

No idea, to be honest. Someone else will have to take care of this, as I do not have access to a Windows build environment.

How to handle multiple static libraries in a package? What if they have circular dependencies?
Static builds may have compiled out some features necessary in a shared build (though it seems this is more for coercing never-shared projects to be shared rather than replacing an intentional link=shared option).

This is just meant as an easy way to convert existing static libraries to shared ones, in case the package in question does not build them (see lua). Instead of patching the package, we can just use this function.

Are soname numbers just skipped? What about library IDs on macOS? I see something is set; is that a full path or just the filename?

Sorry, I skipped them for the initial version. I just pushed an updated version with support for specifying a version.

Other compiler support?

I think it should be pretty easy to extend the function to support additional compilers.

tgamblin

Looks good to me -- I think we can punt on windows for the moment, but there are still a few minor requests.

tgamblin · 2017-11-02T21:09:50Z

lib/spack/spack/test/build_environment.py

+        os.environ['SPACK_TEST_COMMAND'] = 'dump-args'
+
+        expected = {
+            'linux': '/bin/mycc -Wl,-rpath,/spack-test-prefix/lib -Wl,-rpath,/spack-test-prefix/lib64 -shared -Wl,-soname,{2} -Wl,--whole-archive {0} -Wl,--no-whole-archive -o {1}',  # noqa: E501


split across multiple lines (no NOQA)

tgamblin · 2017-11-02T21:11:57Z

lib/spack/spack/test/build_environment.py

+
+class BuildEnvironmentTest(unittest.TestCase):
+
+    def setUp(self):


can you use a pytest fixture for this? We still have some old unittest style tests but we're trying to switch to pytest. You can probably just make a build_environment fixture that does all the same stuff as this, and yields a named tuple with cc, ld, etc. It should probably also unset the environment variables after yielding so as not to interfere with other tests.

tgamblin · 2017-11-02T21:12:26Z

lib/spack/spack/test/build_environment.py

+        if 'SPACK_DEPENDENCIES' in os.environ:
+            del os.environ['SPACK_DEPENDENCIES']
+
+    def test_static_to_shared_library(self):


This test is fine but can be moved to module scope if it's a pytest test. It should take the build_environment fixture as a parameter.

tgamblin · 2017-11-02T21:13:14Z

lib/spack/spack/build_environment.py

+            soname += '.{0}'.format(version)
+
+        compiler_args = [
+            '-shared',


can you add a TODO in here somewhere that says these arguments should really be added to the various compiler classes and generalized for different compilers?

tgamblin · 2017-11-02T21:13:36Z

lib/spack/spack/build_environment.py

+    Converts a static library to a shared library.
+
+    Parameters:
+        static_lib (str): Path to the static library.


There should be a note here or in the description that this library must be built with PIC.

michaelkuhn · 2017-11-02T21:56:54Z

I have pushed a new version. It includes some updates regarding the version handling (if we set a version X.Y.Z, the library is expected to be found at libfoo.so.X.Y.Z, so we create a few symlinks) and should address all of @tgamblin's comments.

tgamblin · 2017-11-02T23:06:39Z

Thanks! LGTM!

michaelkuhn · 2017-11-05T10:15:17Z

I wonder why the CI tests are failing, they work fine locally. Is there something special about the CI environment?

junghans · 2017-11-06T18:13:26Z

lib/spack/spack/test/build_environment.py

+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/llnl/spack


Update url to spack/spack!

The static_to_shared_library function takes an existing static library and produces a shared library based on it.

michaelkuhn · 2017-11-06T20:29:38Z

OK, I think I have found the problem (SPACK_DEBUG_LOG_ID was missing). Should be good to go now. :-)

michaelkuhn force-pushed the static-to-shared branch from 86a8838 to 1f85abe Compare November 2, 2017 01:40

junghans approved these changes Nov 2, 2017

View reviewed changes

tgamblin requested changes Nov 2, 2017

View reviewed changes

michaelkuhn force-pushed the static-to-shared branch from 1f85abe to 5fae802 Compare November 2, 2017 19:57

michaelkuhn force-pushed the static-to-shared branch from 5fae802 to 69f8050 Compare November 2, 2017 20:58

tgamblin requested changes Nov 2, 2017

View reviewed changes

michaelkuhn force-pushed the static-to-shared branch 2 times, most recently from 20c7fb2 to 2786557 Compare November 2, 2017 21:53

michaelkuhn force-pushed the static-to-shared branch from 2786557 to bdbc3e0 Compare November 2, 2017 22:01

tgamblin added linking build-environment labels Nov 2, 2017

tgamblin approved these changes Nov 2, 2017

View reviewed changes

junghans reviewed Nov 6, 2017

View reviewed changes

Introduce static_to_shared_library function

88bd0a7

The static_to_shared_library function takes an existing static library and produces a shared library based on it.

michaelkuhn force-pushed the static-to-shared branch from bdbc3e0 to 88bd0a7 Compare November 6, 2017 19:15

junghans merged commit ca73103 into spack:develop Nov 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce static_to_shared_library function #6092

Introduce static_to_shared_library function #6092

michaelkuhn commented Nov 2, 2017

junghans commented Nov 2, 2017

michaelkuhn commented Nov 2, 2017

junghans commented Nov 2, 2017 •

edited

tgamblin left a comment

michaelkuhn commented Nov 2, 2017

mathstuf commented Nov 2, 2017

michaelkuhn commented Nov 2, 2017

tgamblin left a comment

tgamblin Nov 2, 2017

tgamblin Nov 2, 2017

tgamblin Nov 2, 2017

tgamblin Nov 2, 2017

tgamblin Nov 2, 2017

michaelkuhn commented Nov 2, 2017

tgamblin commented Nov 2, 2017

michaelkuhn commented Nov 5, 2017

junghans Nov 6, 2017

michaelkuhn commented Nov 6, 2017


		class BuildEnvironmentTest(unittest.TestCase):

		def setUp(self):

Introduce static_to_shared_library function #6092

Introduce static_to_shared_library function #6092

Conversation

michaelkuhn commented Nov 2, 2017

junghans commented Nov 2, 2017

michaelkuhn commented Nov 2, 2017

junghans commented Nov 2, 2017 • edited

tgamblin left a comment

Choose a reason for hiding this comment

michaelkuhn commented Nov 2, 2017

mathstuf commented Nov 2, 2017

michaelkuhn commented Nov 2, 2017

tgamblin left a comment

Choose a reason for hiding this comment

tgamblin Nov 2, 2017

Choose a reason for hiding this comment

tgamblin Nov 2, 2017

Choose a reason for hiding this comment

tgamblin Nov 2, 2017

Choose a reason for hiding this comment

tgamblin Nov 2, 2017

Choose a reason for hiding this comment

tgamblin Nov 2, 2017

Choose a reason for hiding this comment

michaelkuhn commented Nov 2, 2017

tgamblin commented Nov 2, 2017

michaelkuhn commented Nov 5, 2017

junghans Nov 6, 2017

Choose a reason for hiding this comment

michaelkuhn commented Nov 6, 2017

junghans commented Nov 2, 2017 •

edited