!include/fcntl.h: align open flags with Linux values

[xiaoxiang781216]

Summary

Align the NuttX open(2) flag constants (O_RDONLY/O_WRONLY/O_RDWR) with the Linux asm-generic values (0/1/2 instead of 1/2/3) so that the FUSE wire protocol and other cross-platform interfaces work without conversion.

• Realign the access-mode flags and the remaining open flags (O_CREAT, O_EXCL, O_APPEND, O_TRUNC, O_NONBLOCK, O_SYNC, O_DSYNC, O_DIRECT, O_DIRECTORY, O_NOFOLLOW, O_NOATIME, O_CLOEXEC, O_PATH, O_TMPFILE, etc.) with the Linux asm-generic values.
• Update all code that used (flags & O_RDONLY) as a bitmask check (which is always 0 now that O_RDONLY == 0) to use (flags & O_ACCMODE) comparisons instead, across fs/, drivers/, libs/, arch/, sched/, net/, and graphics/.
• Update the NUTTX_O_* constants in include/nuttx/fs/hostfs.h to match, and fix the sim hostfs open-flag mapping.

Access-mode flag values across major operating systems

Constant	NuttX (before)	NuttX (after)	Linux	glibc	FreeBSD	macOS
O_RDONLY	1	0	0	0	0	0
O_WRONLY	2	1	1	1	1	1
O_RDWR	3	2	2	2	2	2
O_ACCMODE	3	3	3	3	3	3

Before this change NuttX was the only major OS using 1/2/3 for the access-mode flags; every other platform (Linux, glibc, FreeBSD, macOS) uses the 0/1/2 convention. After this change NuttX matches the de-facto standard, so interfaces that pass open flags directly across the kernel/userspace or host/guest boundary (e.g. FUSE) work without conversion.

Impact

• Compatibility: The new flag values match Linux, so FUSE and other wire protocols that pass open flags directly no longer need translation. This is the primary motivation.
• Behavior: Existing code that performed (flags & O_RDONLY) would silently break (O_RDONLY is now 0), so every such site has been audited and converted to (flags & O_ACCMODE).
• No new APIs: No public API signatures change; only the numeric values of existing #defines change.
• Build: No build-system changes required.

Testing

Tested on the sim:nsh target (Linux x86_64 host, gcc).

Boot log:

NuttShell (NSH) NuttX-10.4.0
nsh> uname -a
NuttX 10.4.0 6bb526f459a Jun 29 2026 23:37:06 sim sim
nsh> ps
  TID   PID  PPID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK
 STACK COMMAND
    0     0     0   0 FIFO     Kthread   - Ready              0000000000000000 0069584 Idle_Task
    1     0     0 224 FIFO     Kthread   - Waiting  Semaphore 0000000000000000 0067456 sim_loop_wq
    2     0     0 224 FIFO     Kthread   - Waiting  Semaphore 0000000000000000 0067472 hpwork
    4     4     0 100 FIFO     Task      - Running            0000000000000000 0067496 nsh_main
nsh> mount
  /bin type binfs
  /data type hostfs
  /etc type romfs
  /proc type procfs
nsh> cat /proc/version
NuttX version 10.4.0 6bb526f459a Jun 29 2026 23:37:06 sim:nsh

OSTest: ran ostest from the NSH prompt — all tests passed with no failures or panics. Final summary:

waitpid_last: PASS: PID 10 waitpid failed with ECHILD.  That may be
waitpid_last: PASS: PID 13 waitpid failed with ECHILD.  That may be
waitpid_last: PASS: PID 19 waitpid failed with ECHILD.  That may be
waitpid_last: PASS: PID 25 waitpid failed with ECHILD.  That may be

[github-actions]

MemBrowse Memory Report

arduino-mega2560

• flash: .text -58 B (-0.1%, 66,353 B / 262,144 B, total: 25% used)

esp32-devkitc

• ROM: .flash.rodata -192 B, .flash.text -716 B (-0.8%, 118,864 B / 4,194,272 B, total: 3% used)
• drom0_0_seg: .flash.rodata -192 B (-1.6%, 12,192 B / 4,194,272 B, total: 0% used)
• irom0_0_seg: .flash.text -716 B (-0.8%, 84,336 B / 3,342,304 B, total: 3% used)

hifive1-revb

• flash: .text +24 B (+0.0%, 80,220 B / 4,194,304 B, total: 2% used)

mirtoo

• kseg0_progmem: .text +60 B (+0.1%, 66,164 B / 131,072 B, total: 50% used)

qemu-armv8a

• Code: .rodata -192 B, .text.arm64_fpu_procfs_open -4 B, .text.cp_handler -4 B, .text.dd_main -388 B, .text.fclose +4 B, .text.fdt_open -4 B, .text.file_mq_timedreceive_internal +8 B, .text.file_mq_timedsend_internal +4 B, .text.file_readv +8 B, .text.file_truncate +4 B, .text.file_vopen +4 B, .text.file_writev +4 B, .text.hostfs_open -4 B, .text.hostfs_write +4 B, .text.inode_checkopenperm +8 B, .text.lib_fflush_unlocked +4 B, .text.lib_flushall +4 B, .text.lib_fwrite_unlocked +4 B, .text.lib_mode2oflags +12 B, .text.mount_open +4 B, .text.nsh_parse_command -4 B, .text.romfs_open -4 B, .text.task_init_info -4 B, .text.tcbinfo_open -4 B, .text.tmpfs_open -8 B, .text.uptime_open -4 B, .text.version_open -4 B (-0.5%, 310,610 B)

qemu-intel64

• Code: .text -1,470 B (-0.0%, 8,651,730 B)
• Data: .rodata -160 B (-0.1%, 118,835 B)

rx65n-rsk2mb

• ROM: .rodata -200 B, .text -768 B (-1.2%, 82,140 B / 2,097,152 B, total: 4% used)

s698pm-dkit

• Code: .text -288 B (-0.1%, 356,848 B)

stm32-nucleo-f103rb

• flash: .text +16 B (+0.0%, 33,224 B / 131,072 B, total: 25% used)

[acassis]

@xiaoxiang781216 shouldn't it be considered a breaking change? Although the symbols still the same their values changes, so some external lib or app (that for some reason copied this old header file) will fail to run new NuttX version.

[xiaoxiang781216]

Yes, it's a breaking change, but it actually improves the compatibility with other major OS:

Constant	NuttX (before)	NuttX (after)	Linux	glibc	FreeBSD	macOS
O_RDONLY	1	0	0	0	0	0
O_WRONLY	2	1	1	1	1	1
O_RDWR	3	2	2	2	2	2
O_ACCMODE	3	3	3	3	3	3

Many 3rd party libraries have to modified to work with NuttX just because NuttX defines O_RDONLY/O_WRONLY/O_RDWR to different values.

[acassis]

Yes, I agree on that!], but the meaning of "breaking changes" is: breaking change with previous NuttX versions. Even when it is to improve the general compatibility.

[xiaoxiang781216]

I added ! to commit message and the breaking change label.