CPU loop running zoauversion -c after upgrading to coreutils-9.5.20240703_180011.zos using altbin

[gngrossi]

[SYSA] bash-5.2$ zoauversion
2024/06/13 14:07:12 CUT v1.3.2.0 edc928b5 5828 PH60438 1232 4da79630

After issuing Ctrl-c, this output is rendered to the screen.

[SYSA] bash-5.2$ zoauversion -c
----- ZOAU diagnostic report -----
Fri Jul 5 15:39:16 CDT 2024

LPAR: SYSA
Hostname: L98MPSYSA.HEWITT.COM
CPC type: 3931
OS: z/OS 03.01

User ID: @02858
uid=2858(@02858) gid=65(@ISCICS1) groups=65(@ISCICS1),29113023($B0TBUNS),400036(@BAGRP01),78(@ISEFTT1),400048(@ISGRP01),400001(@ISHLD),400060(@ISWAST1),64(@ISZCX1),63(@ZCXADM1),9503(CFZUSRGP),9004(IZUUSER),68(ZCONPGRP),67(ZCONTGRP),40(ZCXDGRP)

ulimit:
core file 15842b
cpu time unlimited
data size unlimited
file size unlimited
stack size unlimited
file descriptors 64000
address space unlimited
memory above bar 12288m

Temp dir:

--

[SYSA] bash-5.2$ echo $PATH
/hewitt/zopentools/zopen_repo/usr/local/altbin:/hewitt/zopentools/zopen_repo/usr/local/bin:/hewitt/zopentools/zopen_repo/usr/bin:/hewitt/zopentools/zopen_repo/bin:/hewitt/zopentools/zopen_repo/boot:/usr/lpp/IBM/zoau/v1r3/bin:/usr/lpp/IBM/cyp/v3r12/pyz/bin:/usr/lpp/java/J17.0_64/bin:/usr/lpp/java/J8.0_64/bin:/bin:.:/hewitt/coz/current/bin:/usr/local/bin:/usr/local/sbin:/usr/lpp/ported/bin:/usr/sbin:/usr/lpp/NFS:/u/@02858/scripts:

--

[@02858@SYSA]$ ps -ef
UID PID PPID C STIME TTY TIME CMD
@02858 17498520 84611300 - 15:38:33 ttyp0001 0:00 bash
@02858 17499327 67833987 - 15:16:20 ttyp0000 0:00 bash
@02858 723645 726320 - 15:39:16 ttyp0001 0:07 sh -c zoaversionhelper '-c' </u/@02858/tmp/ZOAU_stdin_gHDaxQ >/u/@02858/tmp/ZOAU_stdout_eEDdxQ 2>/u/@02858/tmp/ZOAU_stderr_AbHBxQ
@02858 34278178 34280817 - 15:39:17 ttyp0001 0:07 df -k /tmp
@02858 17501073 17502665 - 15:39:24 ttyp0002 0:00 ps -ef
@02858 34279496 17499327 - 15:38:33 ttyp0000 0:00 ssh mvs-sysa
@02858 67833987 34278573 - 15:16:20 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R
@02858 84611300 51057970 - 15:38:33 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R
@02858 17502665 84612538 - 15:38:54 ttyp0002 0:00 sh -L
@02858 726320 17498520 - 15:39:16 ttyp0001 0:00 zoauversion -c
@02858 34280817 723645 - 15:39:16 ttyp0001 0:07 /bin/sh /usr/lpp/IBM/zoau/v1r3/bin/zoaversionhelper -c
@02858 84612538 51055655 - 15:38:53 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R

[@02858@SYSA]$ ps -ef
UID PID PPID C STIME TTY TIME CMD
@02858 17498520 84611300 - 15:38:33 ttyp0001 0:00 bash
@02858 17499327 67833987 - 15:16:20 ttyp0000 0:00 bash
@02858 723645 726320 - 15:39:16 ttyp0001 0:11 sh -c zoaversionhelper '-c' </u/@02858/tmp/ZOAU_stdin_gHDaxQ >/u/@02858/tmp/ZOAU_stdout_eEDdxQ 2>/u/@02858/tmp/ZOAU_stderr_AbHBxQ
@02858 34278178 34280817 - 15:39:17 ttyp0001 0:11 df -k /tmp
@02858 34278289 17502665 - 15:39:28 ttyp0002 0:00 ps -ef
@02858 34279496 17499327 - 15:38:33 ttyp0000 0:00 ssh mvs-sysa
@02858 67833987 34278573 - 15:16:20 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R
@02858 84611300 51057970 - 15:38:33 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R
@02858 17502665 84612538 - 15:38:54 ttyp0002 0:00 sh -L
@02858 726320 17498520 - 15:39:16 ttyp0001 0:00 zoauversion -c
@02858 34280817 723645 - 15:39:16 ttyp0001 0:11 /bin/sh /usr/lpp/IBM/zoau/v1r3/bin/zoaversionhelper -c
@02858 84612538 51055655 - 15:38:53 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R

[@02858@SYSA]$ ps -ef
UID PID PPID C STIME TTY TIME CMD
@02858 17498520 84611300 - 15:38:33 ttyp0001 0:00 bash
@02858 17499327 67833987 - 15:16:20 ttyp0000 0:00 bash
@02858 723645 726320 - 15:39:16 ttyp0001 0:15 sh -c zoaversionhelper '-c' </u/@02858/tmp/ZOAU_stdin_gHDaxQ >/u/@02858/tmp/ZOAU_stdout_eEDdxQ 2>/u/@02858/tmp/ZOAU_stderr_AbHBxQ
@02858 34278178 34280817 - 15:39:17 ttyp0001 0:15 df -k /tmp
@02858 51055505 17502665 - 15:39:33 ttyp0002 0:00 ps -ef
@02858 34279496 17499327 - 15:38:33 ttyp0000 0:00 ssh mvs-sysa
@02858 67833987 34278573 - 15:16:20 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R
@02858 84611300 51057970 - 15:38:33 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R
@02858 17502665 84612538 - 15:38:54 ttyp0002 0:00 sh -L
@02858 726320 17498520 - 15:39:16 ttyp0001 0:00 zoauversion -c
@02858 34280817 723645 - 15:39:16 ttyp0001 0:15 /bin/sh /usr/lpp/IBM/zoau/v1r3/bin/zoaversionhelper -c
@02858 84612538 51055655 - 15:38:53 ? 0:00 /usr/sbin/sshd -f /etc/ssh/sshd_config -R

[IgorTodorovskiIBM]

Interesting, I'm not seeing a hang with altbin. This is my output:

2024/06/13 14:07:12 CUT v1.3.2.0 edc928b5 5828 PH60438 1232 4da79630

If you edit zoaversionhelper, there's a comment #set -x on line 12. Can you uncomment and rerun. That should print the offending lines that are hanging. FYI @AnthonyGiorgio

[gngrossi]

@IgorTodorovskiIBM You don't see this output from zoaversion -c ?

[IgorTodorovskiIBM]

Ah, I did not try -c.

zoaversion -c
----- ZOAU diagnostic report -----
Tue Jul  9 09:52:35 EDT 2024

LPAR:     SA2B
Hostname: ZOSCAN2B.pok.stglabs.ibm.com
CPC type: 8561
OS:       z/OS 02.04

User ID: ITODORO
uid=365787(ITODORO) gid=5000006(BASIC) groups=5000006(BASIC),105(CDEV),1021274(OPENZG)

ulimit:
core file         8192b
cpu time          unlimited
data size         unlimited
file size         17179869176b
stack size        unlimited
file descriptors  64000
address space     unlimited
memory above bar  6144m

Temp dir:
Filesystem     1K-blocks     Used Available Use% Mounted on
/tmp            51200000 15282816  35917184  30% /tmp

PATH: /home/itodoro/install_test/zoau13/bin:/home/itodoro/install_test/usr/lpp/IBM/oelcpp/v2r0/bin/:/home/opnzos/local/ibm-clang-driver.org/bin/:/home/itodoro/install_test/go/bin:/home/itodoro/zopen/usr/local/altbin:/home/itodoro/zopen/usr/local/bin:/home/itodoro/zopen/usr/bin:/home/itodoro/zopen/bin:/home/itodoro/zopen/boot:/home/itodoro/zpm/opencpp/1.0.0.0/bin/:/home/itodoro/node-v16:/home/itodoro/node-v16/bin:/home/opnzos/local/bin:/home/itodoro/node-v16/tools/python/bin/:/home/opnzos/local/rocket/bin/:/home/itodoro/node-v16/source/node/install/bin:/home/itodoro/node-v16/source/node/install/bin/:/home/itodoro/.local/bin:/home/opnzos/local/zpm/bin/:/home/opnzos/local/zpm/zosadk/bin:/home/opnzos/local/zoau/bin/:/home/itodoro/node-build/source/node/install/bin:/home/opnzos/local/rocket/bin:/home/itodoro/bin/:/bin:/home/itodoro/bin:/shadow/c390.tools/oe:/bin/:/home/itodoro/scripts:/home/trtobey/bin/:

LIBPATH: /home/itodoro/install_test/zoau13/lib:/home/itodoro/install_test/go/lib:/usr/lib:/home/itodoro/zopen/usr/local/lib:/home/itodoro/zopen/usr/lib:/home/itodoro/node-v16/tools/python/lib/:/lib/lpp/hzc/lib/:/home/opnzos/local/zoau/lib:.:/usr/lib/:

ZOAU build info: 2024/06/13 14:07:12 CUT v1.3.2.0 edc928b5 5828 PH60438 1232 4da79630
ZOAU_HOME dir:

Error: ZOAU_HOME must point to the ZOAU installation directory.

[gngrossi]

Looks like it's the df command...

Used Ctrl-c
[SYSA] bash-5.2$ df -k /tmp

[SYSA] bash-5.2$ /bin/df -k /tmp
Mounted on Filesystem Avail/Total Files Status
/SYSA/tmp (HFS.SYSA.TMP) 3745367/4112640 4294966309 Available

--

[SYSA] bash-5.2$ which df
/hewitt/zopentools/zopen_repo/usr/local/altbin/df

[SYSA] bash-5.2$ df --version
df (GNU coreutils) 9.5
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later https://gnu.org/licenses/gpl.html.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Torbjorn Granlund, David MacKenzie, and Paul Eggert.

--

[SYSA] bash-5.2$ which zoaversion
/usr/lpp/IBM/zoau/v1r3/bin/zoaversion

[SYSA] bash-5.2$ zoaversion -c
----- ZOAU diagnostic report -----
Tue Jul 9 14:47:39 CDT 2024

LPAR: SYSA
Hostname: L98MPSYSA.HEWITT.COM
CPC type: 3931
OS: z/OS 03.01

User ID: @02858
uid=2858(@02858) gid=65(@ISCICS1) groups=65(@ISCICS1),29113023($B0TBUNS),400036(@BAGRP01),78(@ISEFTT1),400048(@ISGRP01),400001(@ISHLD),400060(@ISWAST1),64(@ISZCX1),63(@ZCXADM1),9503(CFZUSRGP),9004(IZUUSER),68(ZCONPGRP),67(ZCONTGRP),40(ZCXDGRP)

ulimit:
core file 15842b
cpu time unlimited
data size unlimited
file size unlimited
stack size unlimited
file descriptors 64000
address space unlimited
memory above bar 12288m

Temp dir:

• ME=zoaversion
• InvalidOption=1
• TooManyParms=2
• build=0
• check=0
• debug=0
• json=0
• verbose=0
• getopts :bcdhjv opt
• check=1
• getopts :bcdhjv opt
• [ 0 -eq 0 ]
• [ 1 -eq 0 ]
• [ 0 -eq 1 ]
• expr 2 - 1
• 1 /u/@02858/tmp/shggChGCGHA
• shift 1
• [ 0 -ne 0 ]
• • mydir /SYSA/tmp/gary/zoaversionhelper
• 1 /u/@02858/tmp/shHhCFaEAhA
• • basename /SYSA/tmp/gary/zoaversionhelper
• 1 /u/@02858/tmp/sheBCGhgfdA
  cmdlinename=zoaversionhelper
• [ zoaversionhelper = /SYSA/tmp/gary/zoaversionhelper ]
• • dirname /SYSA/tmp/gary/zoaversionhelper
• 1 /u/@02858/tmp/shfHhDfBfHA
  mydir=/SYSA/tmp/gary
• echo /SYSA/tmp/gary
  mydir=/SYSA/tmp/gary
• [ 0 -eq 1 ]
• [ 1 -eq 1 ]
• check
• • uname -Ia
• 1 /u/@02858/tmp/sheDhfEBcDA
  uname=z/OS SYSA 01.00 03 3931
• • 1 /u/@02858/tmp/shdACaGAfcA
• echo z/OS SYSA 01.00 03 3931
• awk {print $1}
  OS=z/OS
• • 1 /u/@02858/tmp/shegbbfgEeA
• echo z/OS SYSA 01.00 03 3931
• awk {print $2}
  LPAR=SYSA
• • 1 /u/@02858/tmp/shCCaGEBDEA
• echo z/OS SYSA 01.00 03 3931
• awk {print $3}
• awk {split($0,a,"."); print a[1]}
  VER_MINOR=01
• • 1 /u/@02858/tmp/shfCdGahedA
• echo z/OS SYSA 01.00 03 3931
• awk {print $4}
  VER_MAJOR=03
• • 1 /u/@02858/tmp/shBACBCgeGA
• echo z/OS SYSA 01.00 03 3931
• awk {print $5}
  CPC_TYPE=3931
• • hostname
• 1 /u/@02858/tmp/shfFdFdFFeA
  HOSTNAME=L98MPSYSA.HEWITT.COM
• • buildinfo
• 1 /u/@02858/tmp/shdhDEgcBeA
• buildfile=/SYSA/tmp/gary/.buildinfo
• [ -f /SYSA/tmp/gary/.buildinfo ]
• version=unknown
• timestamp=unknown
• commit=unknown
• build=unknown
• apar=
• combined_build=unknown
• build_sha=unknown
• [ 0 -eq 0 ]
• printf %s v%s unknown unknown
• [ -n unknown ]
• printf %s unknown
• [ -n unknown ]
• printf %s unknown
• [ -n ]
• [ -n unknown ]
• printf %s unknown
• [ -n unknown ]
• printf %s unknown
• printf \n
  BUILDINFO=unknown vunknown unknown unknown unknown unknown
• echo ----- ZOAU diagnostic report -----
• date
• echo
• echo LPAR: SYSA
• echo Hostname: L98MPSYSA.HEWITT.COM
• echo CPC type: 3931
• echo OS: z/OS 03.01
• echo
• echo User ID: @02858
• id
• echo
• echo ulimit:
• ulimit -a
• echo
• echo Temp dir:
• df -k /tmp

[AnthonyGiorgio]

I ran into that earlier today when troubleshooting another issue. I upgraded all the zopen packages on my system, and it seemed to resolve the issue.

[IgorTodorovskiIBM]

Interesting that df is hanging for you. Maybe it's a sysplex thing? You have a sysplex right?

[gngrossi]

@AnthonyGiorgio All of my zopen packages are up to date (zopen list --upgradeable) except for libiconv.

@IgorTodorovskiIBM Yes, we have a sysplex with 12 LPARs. Note each LPAR has its own /tmp/

thanks

[AnthonyGiorgio]

What's strange is that if I run /global/zopen/usr/local/altbin/df /tmp, it hangs, but if I do cd /tmp; /global/zopen/usr/local/altbin/df ., it runs fine. I wonder what the difference in the argument parsing is?

[gngrossi]

For me, it looks like the difference is between /SYSA/tmp and /tmp

[IgorTodorovskiIBM]

Thanks, I can reproduce it on the sysplex machines. I'll remove df for now until it's fixed

[IgorTodorovskiIBM]

I think the culprit is this:

lrwxrwxrwx    1 BPXROOT DEPTD60    12 May 18  2012 tmp -> $SYSNAME/tmp

$SYSNAME has special meaning in z/OS and it's probably getting substituted to null, leading to an infinite loop.

[AnthonyGiorgio]

I also see $VERSION in the root directory of our system. Perhaps we need to implement special handling of certain strings in zoslib? These aren't environment variables in the traditional sense - they don't expand to anything if you ask the shell.

[IgorTodorovskiIBM]

This is the only info I could find - https://www.ibm.com/docs/en/zos/2.4.0?topic=csd-symlink-bpx1sym-bpx4sym-create-symbolic-link-path-name

Looking at Node.js' code, there's a special case for these vars:

     fs.readlink(base, (err, target) => {
        if (!isWindows) seenLinks[id] = target;
        gotTarget(err, target);
        // On z/OS, $ or /$ at the start of a path denotes a parmlib variable,
        // which requires using the platform's native realpath to resolve.
        if (isZOS && (target.charCodeAt(0) === CHAR_DOLLAR_SIGN ||
            target.charCodeAt(0) === CHAR_FORWARD_SLASH &&
            target.charCodeAt(1) === CHAR_DOLLAR_SIGN)) {
          fs.realpath.native(base, (err, resolvedPath) => {
            if (err) return callback(err);

            seenLinks[id] = resolvedPath;
            gotTarget(err, resolvedPath);
          });

It would make sense to provide common solution in zoslib

[IgorTodorovskiIBM]

Ok, I have a fix coming in zoslib. It will substitute the vars correctly.

[ITODOR@PLPSC ~]$ ./realpath /tmp
/AQTS/tmp
[ITODOR@PLPSC ~]$ ./readlink /tmp
/AQTS/tmp
[ITODOR@PLPSC ~]$ ./df /tmp
Filesystem        1K-blocks    Used Available Use% Mounted on
OMVS.ZFS.AQTS.TMP  20412000 1650187  18761813   9% /AQTS/tmp

@ccw-1 I believe this will fix your Git issue when cloning into /tmp as well once it is in