fs/inode: propagate inode search errors

[imeghar2408-max]

Summary

Fix inode_reserve() to propagate inode search errors instead of continuing
inode creation for every negative return from inode_search().

inode_search() returns -ENOENT when the path is valid but the target inode
does not exist yet. Other negative returns indicate an actual lookup or path
error, such as -ENAMETOOLONG.

Previously, inode_reserve() continued processing any negative return. For an
overlong pathname, this could leave the insertion metadata invalid and reach
inode_insert() with a NULL parent, triggering an assertion.

This change allows inode creation to continue only when inode_search() returns
-ENOENT. All other errors are propagated through the existing cleanup path.

Testing

Tested on the NuttX simulator with ProcFS, TMPFS, and ASan enabled.
No ASan error was reported during the tests.

1. Regression test: overlong pathname

Before the fix

Running an overlong mount path:

nsh> mount -t tmpfs tmpfs /tmp/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

triggered:

dump_assert_info: Assertion failed parent != ((void*)0):
at file: inode/fs_inodereserve.c:140
task: nsh_main

The simulator therefore hit the NULL parent assertion in inode_reserve().

After the fix

The same command returned:

nsh> mount -t tmpfs tmpfs /tmp/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
nsh: mount: mount failed: 36

where 36 is ENAMETOOLONG.

No assertion or crash occurred.

2. Valid pathname / normal inode creation

A valid mount was verified:

nsh> mount -t tmpfs tmpfs /a
nsh>

The mount was then confirmed with:

nsh> mount
  /a type tmpfs
  /bin type binfs
  /data type hostfs
  /etc type romfs
  /proc type procfs
  /tmp type tmpfs

This confirms that valid inode creation and mounting still work after the change.

3. Existing inode handling

Created /a/b and then attempted to create it again:

nsh> mkdir /a/b
nsh>
nsh> mkdir /a/b
nsh: /a/b: mkdir failed: 17

where 17 is EEXIST.

This confirms that an existing inode is still handled as an existing target
rather than being treated as a missing inode.

4. Build

The NuttX simulator build completed successfully with CONFIG_SIM_ASAN=y:

Create version.h
LD: nuttx
...
SIM elf with dynamic libs archive in nuttx.tgz
BUILD EXIT CODE: 0

5. Static checks

The following checks passed:

git diff --check
tools/checkpatch.sh -g HEAD

6. PATH_MAX-specific case

A separate runtime test targeting the PATH_MAX limit was attempted, but the
long NSH input could not be delivered reliably in the simulator environment.
Therefore, no PATH_MAX runtime result is claimed here.

[github-actions]

MemBrowse Memory Report

arduino-mega2560

• flash: .text +12 B (+0.0%, 65,104 B / 262,144 B, total: 25% used)

esp32-devkitc

• ROM: .flash.text +12 B (+0.0%, 124,912 B / 4,194,272 B, total: 3% used)
• irom0_0_seg: .flash.text +12 B (+0.0%, 89,116 B / 3,342,304 B, total: 3% used)

hifive1-revb

• flash: .text +8 B (+0.0%, 83,724 B / 4,194,304 B, total: 2% used)

mirtoo

• kseg0_progmem: .text +4 B (+0.0%, 67,756 B / 131,072 B, total: 52% used)

qemu-armv8a

• Code: .text.inode_reserve +16 B (+0.0%, 337,524 B)

qemu-intel64

• Code: .text +10 B (+0.0%, 8,659,724 B)

s698pm-dkit

• Code: .text +16 B (+0.0%, 365,184 B)

stm32-nucleo-f103rb

• flash: .text +4 B (+0.0%, 34,404 B / 131,072 B, total: 26% used)
  No memory changes detected for:
• rx65n-rsk2mb

[imeghar2408-max]

Hi @xiaoxiang781216 and @jerpelea , the CI formatting issues have been addressed and the latest changes have been pushed. Could you please review this PR when you have time? Thank you!

[xiaoxiang781216]

Hi @xiaoxiang781216 and @jerpelea , the CI formatting issues have been addressed and the latest changes have been pushed. Could you please review this PR when you have time? Thank you!

@imeghar2408-max still error, you can run ./tools/checkpatch.sh -g HEAD~...HEAD before update.

[imeghar2408-max]

Hi @xiaoxiang781216 and @jerpelea , the CI formatting issues have been addressed and the latest changes have been pushed. Could you please review this PR when you have time? Thank you!

@imeghar2408-max still error, you can run ./tools/checkpatch.sh -g HEAD~...HEAD before update.

Hi Xiaoxiang, thanks for the suggestion. I updated the commit message and ran ./tools/checkpatch.sh -c -u -m -g ..HEAD locally; all checks pass now. I've pushed the updated commit. Thank you!