Remove 'interactive remove test' from DUB

[wilzbach]

Nearly every PR currently fails due to this

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/activity

See also: dlang/dub#1157

CC @CyberShadow - I think this should fix the Project-Tester and Martin has done a similar hack before ...

[CyberShadow]

• rm test/test/interactive-remove.sh
  rm: cannot remove 'test/test/interactive-remove.sh': No such file or directory

rm -f ?

[PetarKirov]

rm -> rm -f as @CyberShadow mentioned.

[PetarKirov]

echo failed -> how about something more informative, along the lines of: https://stackoverflow.com/a/3055810/1808755

[MartinNowak]

If you mean the issue when there is an unexpected additional dub version, we should figure out why that is lingering around. Each project is using it's own clean dub cache, maybe we should additionally clean at the start of the test though.

[MartinNowak]

Tests that are failing because the registry isn't available should be addressed separately, I could easily run a mirror dub-registry instance on the CI server itself, but would generally prefer if code.dlang.org's reliability improved instead.

[MartinNowak]

This is an instance of the error I meant.
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/master/212/pipeline#step-502-log-343

[wilzbach]

If you mean the issue when there is an unexpected additional dub version, we should figure out why that is lingering around.
Each project is using it's own clean dub cache, maybe we should additionally clean at the start of the test though.

The "fix" for this pressing issue has been lingering too ;-)

dlang/dub#1157

[wilzbach]

This is an instance of the error I meant.
https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/master/212/pipeline#step-502-log-343

Yes this is exactly what this PR tries to circumveit until the DUB "fix" is in (or another solution has been found).
Btw dlang/dub#1156 should help normal humans finding the error easier ;-)

[MartinNowak]

Looks to me as if it boils down to another dub-registry availability issue (e.g. that test fetches 0.9.21),
Further down you'll find more errors https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fdmd/detail/master/212/pipeline/#step-502-log-445.

[MartinNowak]

I'll install a dub-registry mirror in the next few days, should resolve those kinds of issues.

[wilzbach]

I'll install a dub-registry mirror in the next few days, should resolve those kinds of issues.

While this is great, it seems like the issue is independent from the DUB registry:

https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-5514/1/pipeline

[wilzbach]

@ZombineDev

rm -> rm -f as @CyberShadow mentioned.

Hmm I had a stupid typo the first time which I wouldn't have noticed with -f ;-)

echo failed -> how about something more informative, along the lines of: https://stackoverflow.com/a/3055810/1808755

I simply removed this. As with the test removed, DUB's testsuite should pass again :)

[wilzbach]

(force pushing a dummy commit to restart Jenkins due to a transient error in vibe.d ...)

[CyberShadow]

If you mean the issue when there is an unexpected additional dub version, we should figure out why that is lingering around. Each project is using it's own clean dub cache, maybe we should additionally clean at the start of the test though.

If you are not running each project's tests in an isolated environment, then I would be inclined to say that you are Doing It Wrong :) At least set HOME and TMPDIR/TEMP/TMP to a directory that's wiped between projects' tests (Digger and DAutoTest already do this).

[MartinNowak]

Any useful ideas on testing dub without remote services (registry, github) welcome.

At least set HOME and TMPDIR/TEMP/TMP to a directory that's wiped between projects' tests (Digger and DAutoTest already do this).

We do clean dub folders between runs

ci/pipeline.groovy

Line 206 in fce0397

rm -r '${env.WORKSPACE}/distribution'

, but the idea is to cache downloaded sources as that would be quite resource intensive (and more error prone) to fetch everytime.

[CyberShadow]

Any useful ideas on testing dub without remote services (registry, github) welcome.

OK, how about this.

1. We fix/version the tags tested and update them periodically. This can be done as a cronjob - a bot updates the tags, creates a PR with the auto-merge label, if the new tags pass, then all's well and it's merged; otherwise, we notify the project owners that their new tagged version fails on our CI or whatever else is appropriate.
2. We put the full dependencies of all tested packages in version control! It's going to be a lot of code, but it moves the possible-failure point to the creation of the update PR, which is fine.
3. We firewall off all network access in all tested packages. This may require working with the packages' maintainers to make their test suite runnable offline.

This should make running tests totally deterministic as far as third-party services are concerned, leaving only things such as race conditions.

Edit: if we're putting dependencies in git then we might as well put the tested projects themselves instead of just their tags.

[MartinNowak]

Right that would work.
Sounds a bit too rigid on a first though.
One benefit of just hijacking the Travis-CI testing is that it's almost no work to add a new package.

If it's just for failing package downloads, I'd rather just cache github.com in some local proxy.
For code.dlang.org, the problems seems mostly gone since using the local code-mirror.dlang.io.

A local download cache would also eliminate most download overhead, making total clean&reinstall per test a better choice.

[MartinNowak]

If we really start to see github failures we could finish dub's FileSystemPackageSupplier and use that instead.

[wilzbach]

We fix/version the tags tested and update them periodically. This can be done as a cronjob - a bot updates the tags, creates a PR with the auto-merge label, if the new tags pass, then all's well and it's merged; otherwise, we notify the project owners that their new tagged version fails on our CI or whatever else is appropriate.

I like this idea :)
We could even keep thing simple and only update the packages manually every 3 months.

If it's just for failing package downloads, I'd rather just cache github.com in some local proxy.
For code.dlang.org, the problems seems mostly gone since using the local code-mirror.dlang.io.

Well, at some point we have to eat the dogfood and improve our infrastructure, so I guess having to eat our own dogfood isn't that bad and I really hope that multiple mirrors (dlang/dub#1190) and retrying the download (dlang/dub#1198) will start to fix things.

A local download cache would also eliminate most download overhead, making total clean&reinstall per test a better choice.

+1

In any case this turned out to be the wrong issue (and due to the failing dub-registry) as thankfully Martin improved the debugging output vastly:

• silence confusing test output dub#1184
• improve test output dub#1185