dotty.tools.scripting.BashScriptsTests are not running properly, but tests are succeeding

[magnolia-k]

[info] Test run started
[info] Test dotty.tools.scripting.BashScriptsTests.verifyScalaArgs started
osname[linux]
bashExe: [/bin/bash]

/bin/bash: /home/vagrant/dotty/compiler/target/scala-3.0.0/test-classes/scripting/showArgs.sc: Permission denied
[info] Test dotty.tools.scripting.BashScriptsTests.verifyScalacArgs started
osname[linux]
bashExe: [/bin/bash]

/bin/bash: -script: command not found
[info] Test run finished: 0 failed, 0 ignored, 2 total, 0.061s

BashScriptsTests uses the which function provided by ClasspathTests to get the path of scala and scalac commands, but this function depends on the environment variable PATH, so it cannot get the path of the built binary.

As a result, the which function returns an empty string, and BashScriptsTests interprets the following first argument as a command and tries to execute it.

39   val scalacPath = which("scalac")
40   val scalaPath = which("scala")

[dwijnand]

This is in CI (as opposed to just when tests are run locally), so I think it merits a priority bump.

[dwijnand]

@philwalk do you think this is an easy fix?

[magnolia-k]

@philwalk @dwijnand

Looking at the results of the CI run, the path seems to be incorrect

osname[linux]
bashExe: [/usr/bin/bash]
[info] Test run started
[info] Test dotty.tools.scripting.BashScriptsTests.verifyScalaArgs started

/usr/bin/bash: dist/target/pack/bin/scala: No such file or directory
scalacPath[dist/target/pack/bin/scalac]
[info] Test dotty.tools.scripting.BashScriptsTests.verifyScalacArgs started
osname[linux]
bashExe: [/usr/bin/bash]

/usr/bin/bash: dist/target/pack/bin/scalac: No such file or directory
[info] Test run finished: 0 failed, 0 ignored, 2 total, 0.029s

[magnolia-k]

It will not be generated unless dist/pack is executed between compilation(compile) and testing(test).

.github/workflows/ci.yaml

https://github.com/lampepfl/dotty/blob/504b1c2f596a6e3ed21fc6d841322e027b97c1f2/.github/workflows/ci.yaml#L67-L70

[griggt]

Other tests that require dist/pack as a prerequisite are run by ./project/scripts/bootstrapCmdTests:

https://github.com/lampepfl/dotty/blob/601184787508d8fd655f1dc4e48ed19c418fa710/project/scripts/bootstrapCmdTests#L18-L19

I'm not sure if these tests are suitable (or can be adapted) to be run from there also.

[magnolia-k]

If the existence of dist/target/pack/bin/scala determines whether or not tests need to be run, CI will not run BashScriptsTests all the time (since dist/pack is only run in cmdTest after sbt test).

I think that guaranteeing the order in which BashScriptsTests are executed after dist/pack will help stabilize the tests.

[dwijnand]

Moreover I would love to see a change such that the failure is fatal to the test/CI. 🙏🏼