Isolate tests with external server dependencies

[liggitt]

Several tests at several layers (unit, integration, test-cmd.sh, etc) have dependencies on external servers like dockerhub, etc. When these servers have issues, they can block the merge queue.

Possible failures (all from #3619):

--- FAIL: TestImage (0.91s)
    client_test.go:105: unexpected error: error getting X-Docker-Token from openshift/origin-not-found: Get http://index.docker.io/v1/repositories/openshift/origin-not-found/images/: EOF

error: no image or template matched "mysql"
!!! Error in hack/test-cmd.sh:501
  '[ "$(oc new-app mysql --name=db | grep db)" ]' exited with status 1
Call stack:
  1: hack/test-cmd.sh:501 main(...)
Exiting with status 1

--- FAIL: TestRunAll (2.03s)
    newapp_test.go:686: emptyDir volumes: Error mismatch! Expected <nil>, got no image or template matched "mysql": the image "ee2cb077006e3385648619d6adf66d5ac53cfb11c4b2708636c5be0a12b23ff7" in repository "library/mysql" was not found and may have been deleted

Our core tests should be able to run with minimal dependencies on external servers. Tests that depend on other servers should be moved to an extended test suite.

[soltysh]

Totally agree, unit tests should use mock in those areas, since we want to keep test coverage high with basic unit tests. I just run into:

--- FAIL: TestRunBuild (1.02s)
    newapp_test.go:835: successful ruby app generation: Error mismatch! Expected <nil>, got no image or template matched "openshift/ruby-20-centos7": the image "d5de68d8a25674ccfedfb7415b214f73c0de526d15ab08a5b4d43c1e5e96a1cd" in repository "openshift/ruby-20-centos7" was not found and may have been deleted

Generally speaking we should replace external deps with pre-built stuff available in testing env (eg. docker images, github repos, etc.)

[stevekuznetsov]

I just ran into the following errors:
Timeout panics here:

panic: test timed out after 1m0s
goroutine 52 [running]:
testing.func·008()
    /usr/local/go/src/testing/testing.go:681 +0x168
created by time.goFunc
    /usr/local/go/src/time/sleep.go:129 +0x59
goroutine 1 [chan receive]:
testing.RunTests(0xbfb318, 0xf996a0, 0x8, 0x8, 0xc208095b01)
    /usr/local/go/src/testing/testing.go:556 +0xdc3
testing.(*M).Run(0xc20809b2c0, 0x19405e0)
    /usr/local/go/src/testing/testing.go:485 +0xe1
main.main()
    github.com/openshift/origin/pkg/dockerregistry/_test/_testmain.go:112 +0x382

Git error in asset tests:

Running "cdnify:dist" (cdnify) task
Going through dist/404.html, dist/index.html to update script refs
Warning: Failed to execute "git ls-remote --tags --heads git://github.com/jquery/jquery.git", exit code of #128 Use --force to continue.
Aborted due to warnings.

[liggitt]

we can remove the cdnify task

[smarterclayton]

    /data/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/pkg/generate/app/cmd/newapp_test.go:1223 +0xc45
testing.tRunner(0xc20802db00, 0x1e4f8d8)
    /usr/lib/golang/src/testing/testing.go:447 +0x134
created by testing.RunTests
    /usr/lib/golang/src/testing/testing.go:555 +0xd4e
FAIL    github.com/openshift/origin/pkg/generate/app/cmd    60.061s

[deads2k]

@liggitt are you making progress on this? Doesn't really look like it.

Why not squash individual ones as they come up?

[liggitt]

I am not. dockerhub and github are our biggest dependencies, and they are pervasive.

[smarterclayton]

We've moved some to integration. For now, this is unactionable.

[liggitt]

this is biting us for maintenance releases with dependencies on dockerhub versions that don't exist any more

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[soltysh]

/remove-lifecycle stale
/lifecycle frozen

[liggitt]

/unassign