OSH should be smarter about calling execvpe()

[andychu]

execvpe() apparently results in a lot of execve() calls, which other shells don't do. Is this related to caching?

~/git/alpine/abuild$ strace -ff -e fork,execve  -- ~/git/oil/bin/osh -c 'ls /'                              
execve("/home/andy/git/oil/bin/osh", ["/home/andy/git/oil/bin/osh", "-c", "ls /"], [/* 75 vars */]) = 0
execve("/usr/local/sbin/python", ["python", "/home/andy/git/oil/bin/osh", "-c", "ls /"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/local/bin/python", ["python", "/home/andy/git/oil/bin/osh", "-c", "ls /"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/sbin/python", ["python", "/home/andy/git/oil/bin/osh", "-c", "ls /"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/bin/python", ["python", "/home/andy/git/oil/bin/osh", "-c", "ls /"], [/* 75 vars */]) = 0
strace: Process 24347 attached
[pid 24347] execve("/usr/local/sbin/ls", ["ls", "/"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
[pid 24347] execve("/usr/local/bin/ls", ["ls", "/"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
[pid 24347] execve("/usr/sbin/ls", ["ls", "/"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
[pid 24347] execve("/usr/bin/ls", ["ls", "/"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
[pid 24347] execve("/sbin/ls", ["ls", "/"], [/* 75 vars */]) = -1 ENOENT (No such file or directory)
[pid 24347] execve("/bin/ls", ["ls", "/"], [/* 75 vars */]) = 0
bin   cdrom  etc   initrd.img      lib    lib64       media  opt   root  sbin  srv  tmp  var      vmlinuz.old
boot  dev    home  initrd.img.old  lib32  lost+found  mnt    proc  run   snap  sys  usr  vmlinuz

vs.  dash and bash

~/git/alpine/abuild$ strace -ff -e fork,execve  -- bash -c 'ls /'                                           
execve("/bin/bash", ["bash", "-c", "ls /"], [/* 75 vars */]) = 0
execve("/bin/ls", ["ls", "/"], [/* 74 vars */]) = 0
bin   cdrom  etc   initrd.img      lib    lib64       media  opt   root  sbin  srv  tmp  var      vmlinuz.old
boot  dev    home  initrd.img.old  lib32  lost+found  mnt    proc  run   snap  sys  usr  vmlinuz

[icarroll]

I know that at least bash caches executables on $PATH , most likely for exactly this reason.

[andychu]

Yeah I just checked bash and dash. The first time they stat() everything on PATH. Then the second time they know where it is.

I will have to research the cache invalidation though. It seems like there are 2 cases:

1. $PATH is changed. Technically you don't have to invalidate the whole cache, if the cache lists all of /bin /usr/bin/ etc. and some of them are still on $PATH.

2. If any directory in $PATH changes (new programs can be inserted there). Before using the cache I think you have to check the timestamp on the directory to make sure it hasn't changed.

$ PATH=/bin:/usr/bin:_tmp/1 strace -e stat,execve -ff -- dash -c 'prog; prog'
execve("/bin/dash", ["dash", "-c", "prog; prog"], [/* 74 vars */]) = 0
stat("/home/andy/git/alpine/abuild", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/bin/prog", 0x7fff2c3fedc0)       = -1 ENOENT (No such file or directory)
stat("/usr/bin/prog", 0x7fff2c3fedc0)   = -1 ENOENT (No such file or directory)
stat("_tmp/1/prog", {st_mode=S_IFREG|0775, st_size=7, ...}) = 0
strace: Process 27141 attached
[pid 27141] execve("_tmp/1/prog", ["prog"], [/* 74 vars */]) = -1 ENOEXEC (Exec format error)
[pid 27141] execve("/bin/sh", ["/bin/sh", "_tmp/1/prog"], [/* 74 vars */]) = 0
[pid 27141] stat("/home/andy/git/alpine/abuild", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 27141] stat(".", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
1
[pid 27141] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=27141, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 27142 attached
[pid 27142] execve("_tmp/1/prog", ["prog"], [/* 74 vars */]) = -1 ENOEXEC (Exec format error)
[pid 27142] execve("/bin/sh", ["/bin/sh", "_tmp/1/prog"], [/* 74 vars */]) = 0
[pid 27142] stat("/home/andy/git/alpine/abuild", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 27142] stat(".", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
1
[pid 27142] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=27142, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++

dash does the same thing as bash AFAICT:

andy@lisa:~/git/alpine/abuild$ PATH=/bin:/usr/bin:_tmp/1 strace -e stat,execve -ff -- dash -c 'prog; prog'
execve("/bin/dash", ["dash", "-c", "prog; prog"], [/* 74 vars */]) = 0
stat("/home/andy/git/alpine/abuild", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/bin/prog", 0x7fff1eca4590)       = -1 ENOENT (No such file or directory)
stat("/usr/bin/prog", 0x7fff1eca4590)   = -1 ENOENT (No such file or directory)
stat("_tmp/1/prog", {st_mode=S_IFREG|0775, st_size=7, ...}) = 0
strace: Process 27152 attached
[pid 27152] execve("_tmp/1/prog", ["prog"], [/* 74 vars */]) = -1 ENOEXEC (Exec format error)
[pid 27152] execve("/bin/sh", ["/bin/sh", "_tmp/1/prog"], [/* 74 vars */]) = 0
[pid 27152] stat("/home/andy/git/alpine/abuild", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 27152] stat(".", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
1
[pid 27152] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=27152, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 27153 attached
[pid 27153] execve("_tmp/1/prog", ["prog"], [/* 74 vars */]) = -1 ENOEXEC (Exec format error)
[pid 27153] execve("/bin/sh", ["/bin/sh", "_tmp/1/prog"], [/* 74 vars */]) = 0
[pid 27153] stat("/home/andy/git/alpine/abuild", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
[pid 27153] stat(".", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
1
[pid 27153] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=27153, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++

[andychu]

Released with 0.7.pre1: https://www.oilshell.org/release/0.7.pre1/