cpu/native: add host fs access via VFS

[benpicco]

Contribution description

It is often quite handy to access files from Linux on RIOT and the other way round.

With native this is very straightforward as native is just a Linux process.

This adds a VFS driver to access the host file system from native.

Testing procedure

Run e.g. tests/vfs_default

USEMODULE += fs_native

This will create a folder nvm0 in the current working directory that is also accessible from RIOT.

To change the exported directory to the current dir, set

CFLAGS += -DFS_NATIVE_DIR=\"\"

You should now be able to access the directory of the example from withing RIOT:

main(): This is RIOT! (Version: 2023.04-devel-487-gaf076-cpu/native_fs)
mount points:
	/nvm0

data dir: /nvm0
> ls /nvm0
./
Makefile.ci	252 B
nvm0/
main.c	858 B
Makefile	431 B
MEMORY.bin	8388608 B
../
bin/
total 4 files

> vfs r /nvm0/Makefile 64
00000000: 696e 636c 7564 6520 2e2e 2f4d 616b 6566  include ../Makef
00000010: 696c 652e 7465 7374 735f 636f 6d6d 6f6e  ile.tests_common
00000020: 0a0a 5553 454d 4f44 554c 4520 2b3d 2076  ..USEMODULE += v
00000030: 6673 5f64 6566 6175 6c74 0a55 5345 4d4f  fs_default.USEMO

Maybe this should be the default on native instead of littlefs2?

Issues/PRs references

[chrysn]

Looks straightforward enough to me, with minor comments, and no testing yet.

On a remark / future-evolvability level, I hope that we'll on the long run alter our VFS away from the highly libc dependent structs (stat etc) we have now into something suitably small and understand- and implementable (most flags and modes are just ignored by most file systems). How we use POSIX concepts does make this file system very straightforward, but if and when we change the VFS, we'll have to change a lot of implementations anyway, and this would be just one more.

[benpicco]

Shall I squash?

[benpicco]

I added a subfolder inside the native/ directory with the _native_id to prevent clashes when running multiple native instances at once.

To avoid cluttering the directory too much, empty folders are removed when native exits (via unmount).

[chrysn]

As all old comments are addressed, yes, please squash and rebase.

Maybe this should be the default on native instead of littlefs2?

I'd appreciate that. Unless a file system is explicitly configured, it's a toss-up whether the embedded target board's file system would even match what's used on native, so let's rather go for something that's obviously native-only instead of using something that looks like it might be what'd be used on the board but isn't.

_native_id

I'm not so happy with how these paths are composed. Per the original description, -DFS_NATIVE_DIR=\"\" would mount things to the directory I'm in. Now, I have to create a native directory in there, and if I want to prepopulate the file system with any data, I'd have to figure out that I don't have to guess the PID but there's an otherwise unrelated parameter I can set.

I see how multiple running instances can be an issue, especially given that RIOT will not expect file system content to "just change". But then, same is true for MTD based FSes, right?

I don't have a good way forward. In My Ideal World native FS (and probably also MTD) would use file locking to grab its backend, and failure would result in a printed message at startup a la "Unable to mount FS due to concurrent access; consider passing the RIOT_FS_NATIVE environment variable with a different directory to each instance". Would that work for you?

[benpicco]

How about adding an --isolate-fs startup parameter?
Or a CONFIG_NATIVE_ISOLATE_FS?

[chrysn]

Would work for me. Do you think it'd make sense to make the base path also into a runtime configuration (either via argument or environment)?

[benpicco]

That's a bit more tricky as we get that from a XFA and I didn't want native to diverge too much from real boards here.

[chrysn]

Right, because there can be multiple mount points (and the one currently given with `-D` is only the one on the auto mount point). So IIUC the behavior would be: * automount, no argument => `./native` gets mounted * automount, no argument, FS_NATIVE_DIR override to "" => `.` gets mounted * automount, --isolate-fs => `./native/${_native_id}` gets mounted * explicit mount, no argument => whatever is in the `.hostpath` of the struct gets mounted * explicit mount, --isolate-fs => `./${.hostpath}/${_native_id}` gets mounted` Works for me.

[Teufelchen1]

Worked like a charm! Compiled fine with clang & asan.

[Teufelchen1]

Also tested it using #19401 - ofc this also tested #19401 :)

[benpicco]

Rebased again to solve the merge conflict.

[Teufelchen1]

Just nits!

[chrysn]

I've gone through the code, and have some comments. I'll give it a test run in the next iteration.

[chrysn]

I wonder whether it's warranted to have two PATH_MAX buffers around for an operation that can't be GC'd at link time because it's behind the file system driver -- but it's OK, we're on native, nothing in here will eat significant resources of the host anyway.

(If this were targetting embedded, I'd suggest to have static buffers per compilation unit and use them as needed).

[chrysn]

My concerns as well as @Teufelchen1's have been addressed (thus dismissing the review).

LGTM and passes tests, let's ship it.

It said "just nits" and the nits have been picked.

[benpicco]

Thank you for the review!

bors merge

[bors]

Build succeeded:

• Murdock
• static-tests
• tools-build-success