Shared object not found when run through cmd.run

[silenius]

Description
I have a state with a cmd.run command which fail with:

----------                                          
          ID: srv_jetty_module_jndi_ini                
    Function: cmd.run                                                                                                                         
        Name: /usr/local/openjdk8/bin/java -jar start.jar --add-to-start=jndi              
      Result: False                                     
     Comment: Command "/usr/local/openjdk8/bin/java -jar start.jar --add-to-start=jndi" run                    
     Started: 16:50:20.334125                             
    Duration: 148.536 ms                                  
     Changes:                                                     
              ----------                                     
              pid:                                                                                                                            
                  19886                                       
              retcode:                                                                                                                        
                  1
              stderr:                                                                                                                         
                  ld-elf.so.1: Shared object "libjli.so" not found, required by "java"
              stdout:                                    

If I'm running the corresponding command by hand in a shell it works. I'm not sure why the shared object is not found through cmd.run...

The state.show_sls of the corresponding state file is:

    srv_jetty_module_plus_ini:
        ----------
        __env__:
            base
        __sls__:
            geoserver.jetty.modules
        cmd:
            |_
              ----------
              name:
                  /usr/local/openjdk8/bin/java -jar start.jar --add-to-start=plus
            |_
              ----------
              cwd:
                  /usr/local/geoserver/srv/geoserver-2.16.2
            |_
              ----------
              runas:
                  geoserver
            |_
              ----------
              env:
                  |_
                    ----------
                    JAVA_HOME:
                        /usr/local/openjdk8
                  |_
                    ----------
                    JAVA_OPTS:
                        -server -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xmx2048m -Xms2048m -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
            |_
              ----------
              require:
                  |_
                    ----------
                    file:
                        srv_jetty_module_plus_config
            - run
            |_
              ----------
              order:
                  10024

Setup

This is with:

root@geoserver1:/usr/local/geoserver/srv/geoserver-2.16.2 # pw user show geoserver
geoserver:*:35003:35003::0:0:User &:/nonexistent:/usr/sbin/nologin

root@geoserver1:/usr/local/geoserver/srv/geoserver-2.16.2 # ldd /usr/local/openjdk8/bin/java
/usr/local/openjdk8/bin/java:
        libjli.so => /usr/local/openjdk8/bin/../lib/amd64/jli/libjli.so (0x800672000)
        libz.so.6 => /lib/libz.so.6 (0x800683000)
        libthr.so.3 => /lib/libthr.so.3 (0x80069d000)
        libc.so.7 => /lib/libc.so.7 (0x80024a000)

root@geoserver1:/usr/local/openjdk8 # readelf -d /usr/local/openjdk8/bin/java

Dynamic section at offset 0x3028 contains 29 entries:
  Tag                Type                  Name/Value
 0x000000000000001d RUNPATH              Library runpath: [$ORIGIN/../lib/amd64/jli:$ORIGIN/../lib/amd64]
 0x0000000000000001 NEEDED               Shared library: [libjli.so]
 0x0000000000000001 NEEDED               Shared library: [libz.so.6]
 0x0000000000000001 NEEDED               Shared library: [libthr.so.3]
 0x0000000000000001 NEEDED               Shared library: [libc.so.7]
 0x000000000000000e SONAME               Library soname: [lib.so]
 0x0000000000000015 DEBUG                0x0
 0x0000000000000007 RELA                 0x568
 0x0000000000000008 RELASZ               192 (bytes)
 0x0000000000000009 RELAENT              24 (bytes)
 0x000000006ffffff9 RELACOUNT            6
 0x0000000000000017 JMPREL               0x628
 0x0000000000000002 PLTRELSZ             120 (bytes)
 0x0000000000000003 PLTGOT               0x2020
 0x0000000000000014 PLTREL               RELA
 0x0000000000000006 SYMTAB               0x2b8
 0x000000000000000b SYMENT               24 (bytes)
 0x0000000000000005 STRTAB               0x494
 0x000000000000000a STRSZ                208 (bytes)
 0x000000006ffffef5 GNU_HASH             0x420
 0x0000000000000004 HASH                 0x444
 0x000000000000000c INIT                 0x1388
 0x000000000000000d FINI                 0x139c
 0x000000006ffffff0 VERSYM               0x390
 0x000000006ffffffc VERDEF               0x3a4
 0x000000006ffffffd VERDEFNUM            2
 0x000000006ffffffe VERNEED              0x3dc
 0x000000006fffffff VERNEEDNUM           2
 0x0000000000000000 NULL                 0x0

( I don't see a DT_FLAGS though. which means that $ORIGIN wouldn't work ...?)
(also it looks like rtld doesn't actually check DT_FLAGS)

{{ instance }}_jetty_module_{{ module }}_ini:
  cmd.run:
    - name: {{ config.jdk_conf.JAVA_HOME | path_join('bin', 'java') }} -jar start.jar --add-to-start={{ module }}
    - cwd: {{ config.GEOSERVER_HOME }}
    - runas: {{ config.user }}
    - env:
      - JAVA_HOME: {{ config.jdk_conf.JAVA_HOME }}
      - JAVA_OPTS: {{ config.jdk_conf.JAVA_OPTS }}
    - require:
      - file: {{ instance }}_jetty_module_{{ module }}_config

Versions Report
This is with:

root@salt1:/data/formulas/geoserver-formula/geoserver # salt --versions-report
Salt Version:
           Salt: 2019.2.4
 
Dependency Versions:
           cffi: 1.14.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.7 (default, Apr  2 2020, 08:11:04)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 19.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist:   
         locale: UTF-8
        machine: amd64
        release: 12.1-RELEASE-p3
         system: FreeBSD
        version: Not Installed

[silenius]

If I'm removing the:

    - runas: {{ config.user }}

(thus run as root instead of geoserver) it works. However, if I'm adding a login shell and an existing HOME directory to the geoserver system user it doesn't work. Maybe some ENV isn't passed properly?

[silenius]

I tried to modify the state a little bit for debuging:

----------                                                                                                                                                                                                                                                                                  
          ID: srv_jetty_module_jndi_ini
    Function: cmd.run                                                                                                                                                                                                                                                                       
        Name: realpath /usr/local/openjdk8/bin/java && ls -ld /usr/local/openjdk8/bin/../lib/amd64/jli/libjli.so && ldd /usr/local/openjdk8/bin/../lib/amd64/jli/libjli.so && readelf -d /usr/local/openjdk8/bin/java && ldd /usr/local/openjdk8/bin/java -jar start.jar --add-to-start=jndi
      Result: False                    
     Comment: Command "realpath /usr/local/openjdk8/bin/java && ls -ld /usr/local/openjdk8/bin/../lib/amd64/jli/libjli.so && ldd /usr/local/openjdk8/bin/../lib/amd64/jli/libjli.so && readelf -d /usr/local/openjdk8/bin/java && ldd /usr/local/openjdk8/bin/java -jar start.jar --add-to-s
tart=jndi" run                                                                                                                                                                                                                                                                              
     Started: 18:25:33.110071
    Duration: 180.607 ms                                                                                                                                                                                                                                                                    
     Changes:          
              ----------           
              pid:       
                  63532                                                                                                                       
              retcode:                                
                  1                                                                                                                           
              stderr:                                               
                  ldd: -jar: No such file or directory
                  ldd: start.jar: not a dynamic executable
                  ldd: --add-to-start=jndi: No such file or directory                                                                         
              stdout:                                                
                  /usr/local/openjdk8/bin/java                   
                  -rw-r--r--  1 root  wheel  74368 May  1 16:39 /usr/local/openjdk8/bin/../lib/amd64/jli/libjli.so                            
                  /usr/local/openjdk8/bin/../lib/amd64/jli/libjli.so:
                        libz.so.6 => /lib/libz.so.6 (0x800682000)                                                                             
                        libthr.so.3 => /lib/libthr.so.3 (0x80069c000)                                                                         
                        libc.so.7 => /lib/libc.so.7 (0x80024a000)      
                                                                                                                                              
                  Dynamic section at offset 0x3028 contains 29 entries:                                                                       
                    Tag                Type                  Name/Value                                                                       
                   0x000000000000001d RUNPATH              Library runpath: [$ORIGIN/../lib/amd64/jli:$ORIGIN/../lib/amd64]                   
                   0x0000000000000001 NEEDED               Shared library: [libjli.so]                                                        
                   0x0000000000000001 NEEDED               Shared library: [libz.so.6]                                                        
                   0x0000000000000001 NEEDED               Shared library: [libthr.so.3]                                   
                   0x0000000000000001 NEEDED               Shared library: [libc.so.7]
                   0x000000000000000e SONAME               Library soname: [lib.so]   
                   0x0000000000000015 DEBUG                0x0                                                                                
                   0x0000000000000007 RELA                 0x568                                                                              
                   0x0000000000000008 RELASZ               192 (bytes)                                                                        
                   0x0000000000000009 RELAENT              24 (bytes) 
                   0x000000006ffffff9 RELACOUNT            6     
                   0x0000000000000017 JMPREL               0x628      
                   0x0000000000000002 PLTRELSZ             120 (bytes)
                   0x0000000000000003 PLTGOT               0x2020    
                   0x0000000000000014 PLTREL               RELA 
                   0x0000000000000006 SYMTAB               0x2b8      
                   0x000000000000000b SYMENT               24 (bytes)
                   0x0000000000000005 STRTAB               0x494
                   0x000000000000000a STRSZ                208 (bytes)
                   0x000000006ffffef5 GNU_HASH             0x420     
                   0x0000000000000004 HASH                 0x444
                   0x000000000000000c INIT                 0x1388     
                   0x000000000000000d FINI                 0x139c
                   0x000000006ffffff0 VERSYM               0x390
                   0x000000006ffffffc VERDEF               0x3a4 
                   0x000000006ffffffd VERDEFNUM            2     
                   0x000000006ffffffe VERNEED              0x3dc
                   0x000000006fffffff VERNEEDNUM           2    
                   0x0000000000000000 NULL                 0x0   
                  /usr/local/openjdk8/bin/java:                      
                        libjli.so => not found (0)               
                        libz.so.6 => /lib/libz.so.6 (0x80067c000)
                        libthr.so.3 => /lib/libthr.so.3 (0x800696000)
                        libc.so.7 => /lib/libc.so.7 (0x80024a000)
----------

[silenius]

I found the issue.

With "runas"it has "run this as user X" logic which resets the euid and ruid. So it set the ruid and euid correctly, and the egid and group list, just not the rgid but the mismatched rgid is still enough to taint the process.

So it looks like the Salt process is tainted as rgid is not equal to egid and this causes $ORIGIN to be ignored. I think Salstack isn't correctly resetting the rgid somewhere ..?

(obviously it worked fine from the shell because it's untainted there, so the $ORIGIN lookup works to find the library)

[DmitryKuzmenko]

@silenius thank you for reporting and analyzing the issue. It's not a recipe just one more thing to clear up: as I see from the code there are different paths with or without the group argument set. Could with the group Salt uses sudo with a common interface for different platforms (I'm interested in linux vs bsd here). Without it uses native specifics of su commands.
Could you please try to specify the group in your state file and try it one more time?

[silenius]

Hello @DmitryKuzmenko it works if I'm adding a - group: geoserver argument. Sorry I didn't know that it was supported in the state as it isn't mentionned in the documentation (1) (only - runas: is specified).

(1) https://docs.saltstack.com/en/latest/ref/states/all/salt.states.cmd.html#salt.states.cmd.run

[DmitryKuzmenko]

@oeuftete thank you for mentioning that.
@silenius we definitely have to update docs. Thank you for report.
I'm not closing this as duplicate because there're differences between BSD and Linux code. So we want to be sure both issues will gone.

[DmitryKuzmenko]

Also that issue claims the only documentation change. But I think we have to investigate this behavior at all.