tar can fail due to incorrect error code for missing directory

[rick-masters]

Extracting a tar file can fail on Fiwix because tar expects ENOENT for missing intermediate directories but it is getting ENOTDIR instead.

The relevant source code for tar:

https://git.savannah.gnu.org/cgit/tar.git/tree/src/extract.c#n883

This code should return -ENOENT:

Fiwix/fs/namei.c

Lines 129 to 131 in 6e036aa

	/* that's an non-existent directory */
	*d_res = NULL;
	errno = -ENOTDIR;

ENOTDIR is the error code for trying to do something to a directory that is not a directory.

[mikaku]

Do you have a practical example of when tar fails?
I do an intense use of tar command and I have never seen any problem.

Indeed you're right, the error code ENOTDIR looks inappropriate in this context, but I'm curious why I've never experienced any problem.

[rick-masters]

I'm using tar-1.12 to extract a few directories of linux source code:

tar -xzf /sysa/distfiles/linux-5.10.41.tar.gz linux-5.10.41/scripts linux-5.10.41/include linux-5.10.41/arch/x86/include linux-5.10.41/arch/x86/entry
tar: linux-5.10.41/arch/x86/entry: Could not create directory: error 20
tar: Error exit delayed from previous errors

Changing that return code makes it work.
I'll try to come up with a simpler example that works with the tar included in FiwxOS. I'm not finding one right away but I'll keep trying.

By the way, I should mention that I've been able to run live-bootstrap using the Fiwix kernel. It starts from a tiny hex compiler and tons of source code and build up through several dozen packages all the way up to building the linux kernel. However, that took about 28 significant changes to Fiwix including 13 new syscalls. I'll be working to submit those changes to you, although some of them may be specific to live-bootstrap so we'll have to see whether it makes sense for you to accept them. I can maintain a fork if necessary but I'd rather not.

[rick-masters]

Ok, I think I've got a simple way to reproduce the problem:

[(root) ~]# mkdir -p one/two
[(root) ~]# touch one/two/three
[(root) ~]# tar cf one.tar one
[(root) ~]# rm -rf one
[(root) ~]# tar xf one.tar one/two
tar: one/two: Cannot mkdir: Not a directory
tar: Exiting with failure status due to previous errors

[mikaku]

By the way, I should mention that I've been able to run live-bootstrap using the Fiwix kernel. It starts from a tiny hex compiler and tons of source code and build up through several dozen packages all the way up to building the linux kernel.

That's really amazing. I'm very glad to know Fiwix is something useful for your project.

However, that took about 28 significant changes to Fiwix including 13 new syscalls. I'll be working to submit those changes to you, although some of them may be specific to live-bootstrap so we'll have to see whether it makes sense for you to accept them. I can maintain a fork if necessary but I'd rather not.

13 new syscalls? wow! that's a lot. How is that?

Hmm, oriansj told me in the Fiwix channel at Libera.Chat that you are using Musl C library for your project. Why are you using that C library that is so dependent to the Linux kernel? Why are you not using something more open and standard like Newlib C? Perhaps with Newlib C you can do the same with less changes on Fiwix and also to other tools of the project.

Don't get me wrong. I'm not contrary to add those 13 system calls. I hope with these new system calls I'll be able to port more software to FiwixOS. I'd like to have the list of them, just to make me an idea.

Ok, I think I've got a simple way to reproduce the problem:

Yup, I can easily reproduce the problem.
Feel free to submit a PR.

[rick-masters]

I don't know about musl vs Newlib C. To be honest, I've never heard of Newlib C. Those decisions were made long before I joined to work on the kernel bootstrapping. Musl did seem a bit complex but it supports a lot of functionality. I'll have to ask on the the chat. BTW, you'd be very welcome on the chat. There are a lot of folks following the Fiwix work.

The new syscalls were stat64, lstat64, fstat64, readv, writev, mmap2, chown32, madvise (stub), getdents64, fcntl64, utimes, pipe2, and ftruncate.

[mikaku]

I'll have to ask on the the chat. BTW, you'd be very welcome on the chat. There are a lot of folks following the Fiwix work.

I'll be glad to join to the channel and help on questions about Fiwix.

The new syscalls were stat64, lstat64, fstat64, readv, writev, mmap2, chown32, madvise (stub), getdents64, fcntl64, utimes, pipe2, and ftruncate.

I'm a bit concerned on the xxxx64 system calls if they break the code to be run under i386 processors. Besides this, it's strange you can't build software without these system calls. Normally ./configure take care of this and instructs to the tool to use another one.

I mean, I've been able to port very recent software to FiwixOS without theses system calls. I've seen some of these system calls failing in the ./configure messages while building some of the packages that form FiwixOS but the package was built without any problem.

Anyway, the Fiwix kernel tries to keep compatibility with the old Linux 2.0. These system calls come from Linux 2.4, Linux 2.6, or even more. If you are interested to include them, I think it could be interesting to isolate them with a configure option. Something like: CONFIG_SUPPORT_NEWER_SYSCALLS, or even a better name.

[rick-masters]

I can certainly make config options. Even though these calls may have 64-bit data, to support very large files for example, all of these system calls are defined for x86 / 32-bit Linux, although many are indeed introduced in newer versions.

I'm using this reference:
https://chromium.googlesource.com/chromiumos/docs/+/master/constants/syscalls.md#x86-32_bit

They shouldn't cause any problems with i386 processors. I've run the builds with qemu-system-i386 with no problems.

I can take a closer look at whether musl configuration will allow avoiding some of these syscalls but I've looked before and there was no obvious way to do it. Musl is multi-architecture and we're already configuring with --host=i386. It may not have that kind of granularity around syscalls.

I'll submit a PR for the ENOENT fix soon after I take care of some other things.

[fosslinux]

Hey @mikaku, live-bootstrap developer here :)

Just wanted to note why we are using musl for live-bootstrap as opposed to something like newlib (I haven't actually heard of newlib before, but I'm not sure if it really has all these benefits). Originally while we were waiting for a kernel bootstrap I was targeting Linux 2.6+.

• we compile a wide range of software of varying ages (https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst) including things that touch strange parts of libcs (gmp, mpfr, openssl and of course gcc to name a few)
• musl is highly compatible with glibc by-and-large - which particularly matters for significantly older software that uses glibc-isms
• musl's build system is very simple (no autotools) which means we can build it very early in the process

Specifically for newlib it uses autogen and autotools which are incredibly difficult to work around due to their complexity. We would have to handwrite a Makefile for it which is rather tedious work.

This largely means that we encounter significantly less resistance when building random utilities than we would if we used a lesser known alternative libc (newlib, uclibc, for eg).

I see FiwixOS uses newlib, so there is a chance a good amount of software in live-bootstrap would work with newlib, but I don't see significant gain by choosing newlib, apart from libc complexity (some of which we actually rely on regardless).

---

Thank you very much for your work on Fiwix. From when I first saw it, and this has been re-iterated as it has been moved, Fiwix's simplicity and conformity to older standards is incredibly helpful to providing a rather pain-free kernel bootstrap.

[mikaku]

@rick-masters,

They shouldn't cause any problems with i386 processors. I've run the builds with qemu-system-i386 with no problems.

Keep in mind that for QEMU and more systems the x86 architecture starts on the Pentium, and hence it will accept new i486 and i586 processor instructions that an i386 don't has. I'm running QEMU with the option -cpu 486 to reduce as much as possible the support of these new instructions, but is not enough as QEMU don't has the option -cpu 386. That's why I also use other emulators that can bring pure i386 emulation like 86Box, and specially my own real hardware at home.

I can take a closer look at whether musl configuration will allow avoiding some of these syscalls but I've looked before and there was no obvious way to do it. Musl is multi-architecture and we're already configuring with --host=i386. It may not have that kind of granularity around syscalls.

No, don't waste your time on it.

I'll submit a PR for the ENOENT fix soon after I take care of some other things.

Yes, please.

@fosslinux,

Regarding musl and Newlib debate, I see you firmly convinced and comfortable with musl. I don't want to change your mind. If you ever feel curious with Newlib, you have FiwixOS. All software there has been compiled using the toolchain GCC+binutils+Newlib on the system.

Thank you very much for your work on Fiwix. From when I first saw it, and this has been re-iterated as it has been moved, Fiwix's simplicity and conformity to older standards is incredibly helpful to providing a rather pain-free kernel bootstrap.

I really appreciate your words. Thank you very much.