Support non-embedded targets

[Dirbaio]

Currently defmt only works in ELF targets, and requires a custom linker script. Is supporting non-embedded targets in the radar?

• Targets that don't use ELF: Windows, iOS, MacOS
• Targets that do use ELF, but modifying the linker script is not as practical: Linux, Android

I think the most portable way would be to not rely en ELF/linker tricks and have the proc macros hit flat files (or a sqlite DB?) to assign indexes to format strings.

This would be a big change, so if you don't want to do this because the only goal is supporting embedded devices, I understand :) I thought I'd open this issue to confirm.

[Urhengulas]

This certainly sounds like a desirable feature and feasible if someone likes to contribute or fund it.

[DBLouis]

Is it possible to make defmt works on standard x86 linux? It fails to link with the custom linker script. If I don't use the custom linker script readelf still shows a section that contains my log strings. However when running defmt-print on the ELF it fails with: Error: .defmt data not found.

[japaric]

Is it possible to make defmt works on standard x86 linux?

No OS is supported at the moment; that's what this issue is about. What you are describing are the symptoms of several unimplemented things.

It fails to link with the custom linker script.

sounds about right. wouldn't know the exact reason off the top of my head but may be related to Linux binaries being relocatable. Firmware images use absolute addresses for everything (relocation=static in linker/compiler lingo). This ("format string 'interning'" is probably the hardest thing to fix / design / implement. More so on platforms that don't use the ELF format: macOS and Windows.

If I don't use the custom linker script readelf still shows a section that contains my log strings.

(interesting that they "survive" (are not discarded by the linker) the linking process)

However when running defmt-print on the ELF it fails with: Error: .defmt data not found.

without the custom linker script the log strings won't end in the right place so the .defmt linker section is not created hence the error message

---

The other thing that's missing is a transport crate for Linux and other OSes. That crate defines how the encoded/compressed defmt data is output (e.g. framing) and to where (e.g. stdout, a file, a named pipe, a socket, etc.). You would then pass this defmt data to defmt-print

[DBLouis]

Is there any hope of making it work by setting the right compiler flags so that the log strings are "at the right place"? I don't know much about linking so maybe I am missing the point

[japaric]

maybe? I'm familiar with the details of static linking but not the details of dynamic / relocatable linking so I can't answer that

[DBLouis]

With the following parameters linking succeed but the program segfault immediately.
I think the default defmt script might overwrite stuff.

  -C relocation-model=static  -C target-feature=+crt-static  -C link-arg=-Tdefmt.x

---

For what I understand, the INFO section should not interfere with the loaded program and only be present in the ELF file. If I remove only the last link argument, my program starts, if I put it back it crashes before main.

[DBLouis]

I had assume that the default linker script was always passed as a parameter. Looks like is it not. After adding it to .cargo/config my program compiles and runs, and defmt-print decode frames correctly.

[target.x86_64-unknown-linux-musl]
rustflags = [
  "-C", "target-feature=+crt-static",
  "-C", "relocation-model=static",
  "-C", "link-arg=-T/usr/x86_64-linux-musl/usr/lib/ldscripts/elf_x86_64.x",
  "-C", "link-arg=-Tdefmt.x"
]

It seems to works fine without +crt-static but not without relocation-model=static. For both musl and glibc.

[Urhengulas]

It seems to works fine without +crt-static but not without relocation-model=static. For both musl and glibc.

@DBLouis, very cool that you made it work! 🎉 What is your usecase that makes you want to use defmt over log?

[andresv]

Use case can be that there is a no_std lib that is primarily meant for embedded devices and it has bunch of defmt prints in there and you wanna use this lib also on x86.

It seems one way to do it is just to add different feature flags to the lib itself: https://github.com/embassy-rs/embassy/blob/master/embassy/src/fmt.rs

[newAM]

Use case can be that there is a no_std lib that is primarily meant for embedded devices and it has bunch of defmt prints in there and you wanna use this lib also on x86.

That would work, but I think the goal is to have binary logging on hosted targets, for things like embedded linux where there are storage limitations with normal logging.

[gauteh]

Came across this while working on a defmt-serial target: https://github.com/gauteh/defmt-serial. Wrote an example for the host environment, works with: #463 (comment):

❯ DEFMT_LOG=debug cargo run
   Compiling defmt-serial v0.1.0 (/home/gauteh/dev/embedded/defmt-serial)
   Compiling example-std v0.1.0 (/home/gauteh/dev/embedded/defmt-serial/example-std)
    Finished dev [unoptimized + debuginfo] target(s) in 0.54s
     Running `bash defmt-print-runner.sh target/x86_64-unknown-linux-musl/debug/example-std`
Hello, world!
Logging to info with defmt..
Good bye.
INFO  Hello defmt-world!
└─ example_std::main @ src/main.rs:28
DEBUG Now at: 0
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 1
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 2
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 3
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 4
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 5
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 6
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 7
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 8
└─ example_std::main @ src/main.rs:31
...
DEBUG Now at: 45
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 46
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 47
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 48
└─ example_std::main @ src/main.rs:31
DEBUG Now at: 49
└─ example_std::main @ src/main.rs:31
WARN  Done!
└─ example_std::main @ src/main.rs:34
^C
defmt-serial/example-std on  main via 🦀 v1.61.0-nightly

[Urhengulas]

That is great to see @gauteh! Is there anything we can do to support non-embedded targets better?

[gauteh]

@Urhengulas , I guess the biggest problem is the rustflags that are currently needed. Couldn't get it to work on -gnu target. The rest is solvable:

• target for defmt-outputting directly to stdout without going through defmt-print
• defmt-serial is essentially using a proxy function to output the data, if this is sound, it seems like a general pattern for easily defining a callback that works as a defmt target.

[nathaniel-brough]

If anyone is working on this, you might find the linker scripts and tooling in the google/pigweed/pw_tokeniser directory useful. It's all written in c++ but works for host/embedded-arm but uses similar mechanisms to what defmt uses. i.e. pw_tokeniser and defmt offer very similar functionality.

[elpiel]

Hello,
I saw this issue and since I don't see much work around it after 2 years I was wondering if there's a way to circumvent defmt logging all together?

Right now #463 (comment) seems to be working for linux -gnu as well, the only problem I'm encountering is that doc tests started failing.
I'm using defmt_serial with an StdoutSerial impl from the std example.

note: /usr/bin/ld: /mnt/Disk/Projects/lechev.space/netbox/netbox-hardware/target/x86_64-unknown-linux-gnu/debug/deps/libnetbox_firmware-5b6d3b1c9ed15201.rlib(netbox_firmware-5b6d3b1c9ed15201.14bxe1s9urqapwuu.rcgu.o):
relocation R_X86_64_32 against symbol `rust_eh_personality' can not be used when making a PIE object;
recompile with -fPIE
          /usr/bin/ld: failed to set dynamic section sizes: bad value
          collect2: error: ld returned 1 exit status