Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extern crate loading #1362

Merged
merged 3 commits into from
Jul 15, 2022
Merged

extern crate loading #1362

merged 3 commits into from
Jul 15, 2022

Conversation

philberty
Copy link
Member

@philberty philberty commented Jul 6, 2022

WIP still trying to write dejagnu to enable automated tests

Extern crates statements to tell the front-end to look for another library.
The mechanism here is heavily inspired from gccgo, so when we compile a
library for example we invoke:

  gccrs -g -O2 -frust-crate=mylib -c src/lib.rs -o src/mylib.o

All going well this object file will now contain extra data inside
.rust-export section inside the object file which will be preserved inside
archives and shared objects. When we have another application which uses
this library 'mylib'.

  extern crate mylib;
  use mylib::foo;

  fn main() {
    foo();
  }

We compile using:

gcc -g -O2 -frust-crate=test -c src/main.rs -o src/main.o

When the extern crate line is hit the front-end will look for mylib.o,
libmylib.a, mylib.rox. If it finds a raw object file it will read the
.rust-export section directly from the object for the public metadata
such as public functions, types constants etc. If it fails to find an
object it might find .rox which is the objdump of the .rust-export to a
raw file, it might even find libmylib.a and read the export directly out
of the archive file reusing code from gccgo to do so.

The full compiler pipeline is reused here, so the metatadata is actually
just real rust code. The benifit here is that Rust supports exporting,
macros and generics so this requires the name-resolution and type info
all to be generated and inserted into the apropriate context classes. Since
the metadata is real rust code it means we can reuse the full pipeline to
generate the code as nessecary. So for the simple case of a public struct
we simply emit the AST dump of this struct directly into the metadata. If
its a non-generic public function we emit and extern rust abi block for
that function. If its a trait we can simply emit the trait with the public
memebers. Generics are more complicated since we need to emit the function
fully for it to be compiled correctly this still needs tests to be added.
The hardest part is non generic impl blocks which is still a WIP.

To finally link the two crates together you run:

gcc -g -O2 -o rust-program.exe src/main.o src/mylib.o

Fixes: 1169

@philberty philberty added this to the Imports and visibility milestone Jul 6, 2022
@philberty philberty self-assigned this Jul 6, 2022
@philberty philberty force-pushed the phil/metadata-output branch 9 times, most recently from 29c6c3d to e3b3a33 Compare July 8, 2022 18:34
@philberty philberty changed the title WIP extern crate loading extern crate loading Jul 8, 2022
@philberty philberty linked an issue Jul 14, 2022 that may be closed by this pull request
@philberty
Copy link
Member Author

It seems as though mac does not like dumping data into a custom section. This might need us to just always export a seperate file instead.

This allows AST dump of attributes on Traits this code needs to be applied
for all attribute dumps. Traits also have an implicit Self generic param
this ensure we ignore this during an AST dump.
Extern crates statements to tell the front-end to look for another library.
The mechanism here is heavily inspired from gccgo, so when we compile a
library for example we invoke:

  gccrs -g -O2 -frust-crate=mylib -c src/lib.rs -o src/mylib.o

All going well this object file will now contain extra data inside
.rust-export section inside the object file which will be preserved inside
archives and shared objects. When we have another application which uses
this library 'mylib'.

  extern crate mylib;
  use mylib::foo;

  fn main() {
    foo();
  }

We compile using:

  gcc -g -O2 -frust-crate=test -c src/main.rs -o src/main.o

When the extern crate line is hit the front-end will look for mylib.o,
libmylib.a, mylib.rox. If it finds a raw object file it will read the
.rust-export section directly from the object for the public metadata
such as public functions, types constants etc. If it fails to find an
object it might find .rox which is the objdump of the .rust-export to a
raw file, it might even find libmylib.a and read the export directly out
of the archive file reusing code from gccgo to do so.

The full compiler pipeline is reused here, so the metatadata is actually
just real rust code. The benifit here is that Rust supports exporting,
macros and generics so this requires the name-resolution and type info
all to be generated and inserted into the apropriate context classes. Since
the metadata is real rust code it means we can reuse the full pipeline to
generate the code as nessecary. So for the simple case of a public struct
we simply emit the AST dump of this struct directly into the metadata. If
its a non-generic public function we emit and extern rust abi block for
that function. If its a trait we can simply emit the trait with the public
memebers. Generics are more complicated since we need to emit the function
fully for it to be compiled correctly this still needs tests to be added.
The hardest part is non generic impl blocks which is still a WIP.

To finally link the two crates together you run:

  gcc -g -O2 -o rust-program.exe src/main.o src/mylib.o

Fixes: 1169
This interface will allow us to write the metadata directly to a file.
MacOs does not support metadata embeded directly into a .section of
object files and archives. This allows us to seperate this mechanism so
that we can follow rustc and write this directly to a seperate file.

This patch adds two new options to control metadata outputs

 -frust-embed-metadata this toggles to embed the metadata into .rust_export
 section of the target asm output

 -frust-metadata-output= specifies the path to directly write the metadata
 to file

We need these options as embeding the metadata does not seem to be
supported for all platforms like MacOs so writing directly to file is a
platform agnostic solution.

There is an implicit naming convetion to metadata export files. They need
to be of crate_name.rox. This patch adds some simple validation by checking
the basename of the specified path.
@philberty
Copy link
Member Author

bors r+

@bors
Copy link
Contributor

bors bot commented Jul 15, 2022

Build succeeded:

@bors bors bot merged commit fd2bd65 into master Jul 15, 2022
@philberty philberty deleted the phil/metadata-output branch July 17, 2022 20:04
tschwinge added a commit that referenced this pull request Jul 20, 2022
The recent commit 418aef1
"Support extern crate" (merged in
commit fd2bd65 "Merge #1362"), added
'gcc/testsuite/rust/link/link.exp', which I understand to be a hacked up
variant of the LTO multi-file testing harness (not reviewed in detail).

Since that commit, there are issues in parallel testing, so that, for example
with:

    $ make -j12 check-rust RUNTESTFLAGS='--target_board=unix\{,-m32,-mx32\}'

..., the first variant does test as expected:

    @@ -6490,9 +6490,31 @@ PASS: rust/execute/torture/trait9.rs   -Os   at line 13 (test for warnings, line
     PASS: rust/execute/torture/trait9.rs   -Os  (test for excess errors)
     PASS: rust/execute/torture/trait9.rs   -Os  execution test
     PASS: rust/execute/torture/trait9.rs   -Os  output pattern test
    +Running /home/thomas/tmp/source/gcc/rust/build/trace-slim/source-gcc/gcc/testsuite/rust/link/link.exp ...
    +PASS: rust/link/generic_function_0.rs   -O0  (test for excess errors)
    +[...]
    +PASS: rust/link/trait_import_0.rs   -Os  (test for excess errors)
                    === rust Summary for unix ===

    -# of expected passes           6445
    +# of expected passes           6466
     # of expected failures         31

..., but we then get undesired changes for the second and following variants:

     Running target unix/-m32
     Running /home/thomas/tmp/source/gcc/rust/build/trace-slim/source-gcc/gcc/testsuite/rust/compile/compile.exp ...
     PASS: rust/compile/abi-options1.rs  at line 2 (test for errors, line 1)
     PASS: rust/compile/abi-options1.rs  at line 7 (test for errors, line 6)
     PASS: rust/compile/abi-options1.rs (test for excess errors)
    -PASS: rust/compile/array3.rs  at line 2 (test for warnings, line 1)
    -PASS: rust/compile/array3.rs (test for excess errors)
    -PASS: rust/compile/array_empty_list.rs  at line 3 (test for errors, line 2)
    -PASS: rust/compile/array_empty_list.rs (test for excess errors)
    -PASS: rust/compile/arrays1.rs  (test for errors, line 3)
    -PASS: rust/compile/arrays1.rs (test for excess errors)
     PASS: rust/compile/arrays2.rs  at line 4 (test for errors, line 3)
     PASS: rust/compile/arrays2.rs (test for excess errors)
     PASS: rust/compile/attr-mismatch-crate-name.rs (test for excess errors)
    @@ -8355,110 +8371,215 @@ PASS: rust/compile/torture/generics17.rs   -O3 -g  (test for excess errors)
     PASS: rust/compile/torture/generics17.rs   -Os   at line 18 (test for warnings, line 17)
     PASS: rust/compile/torture/generics17.rs   -Os  (test for excess errors)
     PASS: rust/compile/torture/generics18.rs   -O0   at line 19 (test for warnings, line 18)
    +PASS: rust/compile/torture/generics18.rs   -O0   at line 19 (test for warnings, line 18)
    +PASS: rust/compile/torture/generics18.rs   -O0   at line 5 (test for warnings, line 4)
     PASS: rust/compile/torture/generics18.rs   -O0   at line 5 (test for warnings, line 4)
     PASS: rust/compile/torture/generics18.rs   -O0  (test for excess errors)
    +PASS: rust/compile/torture/generics18.rs   -O0  (test for excess errors)

..., and so on.

This has been determined to be due to repetitive 'runtest_file_p' calls (it's
a mess...), thusly fixed (some would call it a "hack", no doubt), and verified
that, for example:

    $ make -j12 check-rust RUNTESTFLAGS='--target_board=unix\{,-m32,-mx32\} link.exp=simple_function_0.rs'

... only runs the one test as indicated.
bors bot added a commit that referenced this pull request Jul 20, 2022
1404: 'gcc/testsuite/rust/link/link.exp': Fix 'runtest_file_p' usage r=philberty a=tschwinge

The recent commit 418aef1
"Support extern crate" (merged in
commit fd2bd65 "Merge #1362"), added
'gcc/testsuite/rust/link/link.exp', which I understand to be a hacked up
variant of the LTO multi-file testing harness (not reviewed in detail).

Since that commit, there are issues in parallel testing, so that, for example
with:

    $ make -j12 check-rust RUNTESTFLAGS='--target_board=unix\{,-m32,-mx32\}'

..., the first variant does test as expected:

    `@@` -6490,9 +6490,31 `@@` PASS: rust/execute/torture/trait9.rs   -Os   at line 13 (test for warnings, line
     PASS: rust/execute/torture/trait9.rs   -Os  (test for excess errors)
     PASS: rust/execute/torture/trait9.rs   -Os  execution test
     PASS: rust/execute/torture/trait9.rs   -Os  output pattern test
    +Running /home/thomas/tmp/source/gcc/rust/build/trace-slim/source-gcc/gcc/testsuite/rust/link/link.exp ...
    +PASS: rust/link/generic_function_0.rs   -O0  (test for excess errors)
    +[...]
    +PASS: rust/link/trait_import_0.rs   -Os  (test for excess errors)
                    === rust Summary for unix ===

    -# of expected passes           6445
    +# of expected passes           6466
     # of expected failures         31

..., but we then get undesired changes for the second and following variants:

     Running target unix/-m32
     Running /home/thomas/tmp/source/gcc/rust/build/trace-slim/source-gcc/gcc/testsuite/rust/compile/compile.exp ...
     PASS: rust/compile/abi-options1.rs  at line 2 (test for errors, line 1)
     PASS: rust/compile/abi-options1.rs  at line 7 (test for errors, line 6)
     PASS: rust/compile/abi-options1.rs (test for excess errors)
    -PASS: rust/compile/array3.rs  at line 2 (test for warnings, line 1)
    -PASS: rust/compile/array3.rs (test for excess errors)
    -PASS: rust/compile/array_empty_list.rs  at line 3 (test for errors, line 2)
    -PASS: rust/compile/array_empty_list.rs (test for excess errors)
    -PASS: rust/compile/arrays1.rs  (test for errors, line 3)
    -PASS: rust/compile/arrays1.rs (test for excess errors)
     PASS: rust/compile/arrays2.rs  at line 4 (test for errors, line 3)
     PASS: rust/compile/arrays2.rs (test for excess errors)
     PASS: rust/compile/attr-mismatch-crate-name.rs (test for excess errors)
    `@@` -8355,110 +8371,215 `@@` PASS: rust/compile/torture/generics17.rs   -O3 -g  (test for excess errors)
     PASS: rust/compile/torture/generics17.rs   -Os   at line 18 (test for warnings, line 17)
     PASS: rust/compile/torture/generics17.rs   -Os  (test for excess errors)
     PASS: rust/compile/torture/generics18.rs   -O0   at line 19 (test for warnings, line 18)
    +PASS: rust/compile/torture/generics18.rs   -O0   at line 19 (test for warnings, line 18)
    +PASS: rust/compile/torture/generics18.rs   -O0   at line 5 (test for warnings, line 4)
     PASS: rust/compile/torture/generics18.rs   -O0   at line 5 (test for warnings, line 4)
     PASS: rust/compile/torture/generics18.rs   -O0  (test for excess errors)
    +PASS: rust/compile/torture/generics18.rs   -O0  (test for excess errors)

..., and so on.

This has been determined to be due to repetitive 'runtest_file_p' calls (it's
a mess...), thusly fixed (some would call it a "hack", no doubt), and verified
that, for example:

    $ make -j12 check-rust RUNTESTFLAGS='--target_board=unix\{,-m32,-mx32\} link.exp=simple_function_0.rs'

... only runs the one test as indicated.


Co-authored-by: Thomas Schwinge <thomas@codesourcery.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ExternCrate and UseDeclaration do not get resolved in ResolveItem
1 participant