Helix reporting fails due to non-POSIX compliant line in Helix reporter run

[hoyosjs]

• This issue is blocking
• This issue is causing unreasonable pain

Helix reporting fails due to this line in run.sh not being POSIX compliant and -T not being a valid option on BSD mv (macOS):

arcade/src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.sh

Line 17 in 9131f07

mv -T $TMP_ENV_PATH $ENV_PATH

The uploading of testResults.xml succeeds, but the test reporting to AzDO fails. Only the Job node shows up as failed.

Example:
Agent: dci-mac-build-134
Job : b5c4d0e-515d-4c1b-8382-9f6c2add90b2
Workitem: DiagnosticTests

+ mv -T /Users/helix-runner/.azdo-env-tmp /Users/helix-runner/.azdo-env
mv: illegal option -- T
usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory
+ _OLD_PYTHONPATH=
+ export PYTHONPATH=
+ PYTHONPATH=
+ /Users/helix-runner/.azdo-env/bin/python -c 'import azure.devops'
/tmp/helix/working/B65F09CB/p/reporter/run.sh: line 24: /Users/helix-runner/.azdo-env/bin/python: No such file or directory

[hoyosjs]

Looks like the correct fix is to use -h for Darwin systems branching on uname -s.

[jonfortescue]

whoa. @alexperovich have we noticed this previously? This implies to me that we haven't been reporting results from Macs for like 6 months or so

[alexperovich]

I haven't noticed this before. That is really odd that is it now failing. I 100% saw this working on a mac when testing it.

[hoyosjs]

This definitely works for most cases. The runtime definitely gets updates and the tests get reported. I'm not sure if this build agent has a different version of mv (maybe other machines have a brew installed one). We run on OSX.1014.Amd64. Is there anyone else testing in internal? I found aspnet is but their variables still show .vsts-env for the virtualenv. The script is still using -T but it's a different agent (129). Is it maybe different mv getting used?

Also, looks like -h and -T are not available on macOS's BSD mv. Maybe the machines have a brew variant installed (say like a coreutils variant)?

[hoyosjs]

@alexperovich, anything we can do to work around this and #5013. We have no reporting for at least 3 legs with this.

[alexperovich]

I can't think of any easy workaround for this issue. You could change the file in the nuget package after restore, but that wouldn't work well in a build.

[hoyosjs]

@alexperovich any advice on how to proceed here? Reviewing tests on OSX is getting tedious. All our CI and PR builds look like this: https://dev.azure.com/dnceng/internal/_build/results?buildId=563039&_a=summary&view=results. We lose history and given that this is out of band testing history can often help with regression investigation.

cc: @dotnet/dotnet-diag

[ChadNedzlek]

Can we just change the top to /bin/bash?

[hoyosjs]

Do you install a special version of bash on the macOS machines? By default these are not in BSD's bash 3.2.

[ChadNedzlek]

Did we only recently stop installing python to the path? And therefore triggered this if statement for the first time ever? I know @MattGal has been dealing with some fallout from that in other queues.

[alexperovich]

That could cause this. If the newer macs don't have this directory populated then we would start hitting this case.

[ChadNedzlek]

This is super weird that we are only noticing this now. Do we know when (if ever) this used to work? The code is 6 months old. :-( Unfortunately, because this is all done in code in arcade, we don't have good telemetry on this in Helix (because as far as Helix is concerned, everything is fine... we ran the workitem we where given)

[alexperovich]

When I made #4617 it was working on OSX. I specifically tested it there because that is where it was failing.