New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run mock inside a cockpit test machine #3138

Closed
wants to merge 21 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@mvollmer
Member

mvollmer commented Nov 10, 2015

No description provided.

@mvollmer mvollmer changed the title from WIP - Test mock in vm to WIP - Run mock inside a cockpit test machine Nov 10, 2015

@mvollmer mvollmer added the needswork label Nov 10, 2015

Show outdated Hide outdated tools/cockpit.spec
@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 10, 2015

Member

Should vm-build call make-srpm? I'd say yes.

Member

mvollmer commented Nov 10, 2015

Should vm-build call make-srpm? I'd say yes.

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 10, 2015

Member

Should vm-build call make-srpm? I'd say yes.

Done.

Member

mvollmer commented Nov 10, 2015

Should vm-build call make-srpm? I'd say yes.

Done.

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 12, 2015

Member

From check-kubernetes on Fedora 23:

marras 12 20:57:43 localhost.localdomain docker[1000]: time="2015-11-12T13:57:43.256675485-05:00" level=info msg="GET /version"
marras 12 20:57:43 localhost.localdomain docker[1000]: 2015/11/12 13:57:43 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
marras 12 20:57:43 localhost.localdomain docker[1000]: goroutine 1266 [running]:
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.(*conn).serve.func1(0xc8201344d0, 0x7fdbf12808e8, 0xc820024650)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1287 +0xb5 fp=0xc8200454c8 sp=0xc8200453f8
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.call32(0x0, 0x1287be8, 0xc82006c170, 0x1800000018)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/asm_amd64.s:437 +0x3e fp=0xc8200454f0 sp=0xc8200454c8
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.gopanic(0xf0f880, 0xc82000e070)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/panic.go:423 +0x4e9 fp=0xc820045570 sp=0xc8200454f0
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.panicmem()
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/panic.go:42 +0x49 fp=0xc820045598 sp=0xc820045570
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.sigpanic()
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/sigpanic_unix.go:24 +0x2ba fp=0xc8200455e8 sp=0xc820045598
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/docker/docker/api/server.getpwuid(0xffffffff, 0x0, 0x0, 0x0, 0x0)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/credentials_linux.go:75 +0x1f6 fp=0xc820045678 sp=0xc8200455e8
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/docker/docker/api/server.(*Server).LogAction(0xc8201903c0, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620, 0x0, 0x0)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/credentials_linux.go:172 +0xaac fp=0xc820045850 sp=0xc820045678
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/docker/docker/api/server.(*Server).makeHttpHandler.func1(0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/server.go:1680 +0x8aa fp=0xc820045aa8 sp=0xc820045850
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.HandlerFunc.ServeHTTP(0xc820180960, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1422 +0x3a fp=0xc820045ac8 sp=0xc820045aa8
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc8202019a0, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e fp=0xc820045be0 sp=0xc820045ac8
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.serverHandler.ServeHTTP(0xc820388120, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1862 +0x19e fp=0xc820045c40 sp=0xc820045be0
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.(*conn).serve(0xc8201344d0)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1361 +0xbee fp=0xc820045f98 sp=0xc820045c40
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.goexit()
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820045fa0 sp=0xc820045f98
marras 12 20:57:43 localhost.localdomain docker[1000]: created by net/http.(*Server).Serve
Member

mvollmer commented Nov 12, 2015

From check-kubernetes on Fedora 23:

marras 12 20:57:43 localhost.localdomain docker[1000]: time="2015-11-12T13:57:43.256675485-05:00" level=info msg="GET /version"
marras 12 20:57:43 localhost.localdomain docker[1000]: 2015/11/12 13:57:43 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
marras 12 20:57:43 localhost.localdomain docker[1000]: goroutine 1266 [running]:
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.(*conn).serve.func1(0xc8201344d0, 0x7fdbf12808e8, 0xc820024650)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1287 +0xb5 fp=0xc8200454c8 sp=0xc8200453f8
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.call32(0x0, 0x1287be8, 0xc82006c170, 0x1800000018)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/asm_amd64.s:437 +0x3e fp=0xc8200454f0 sp=0xc8200454c8
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.gopanic(0xf0f880, 0xc82000e070)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/panic.go:423 +0x4e9 fp=0xc820045570 sp=0xc8200454f0
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.panicmem()
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/panic.go:42 +0x49 fp=0xc820045598 sp=0xc820045570
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.sigpanic()
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/sigpanic_unix.go:24 +0x2ba fp=0xc8200455e8 sp=0xc820045598
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/docker/docker/api/server.getpwuid(0xffffffff, 0x0, 0x0, 0x0, 0x0)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/credentials_linux.go:75 +0x1f6 fp=0xc820045678 sp=0xc8200455e8
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/docker/docker/api/server.(*Server).LogAction(0xc8201903c0, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620, 0x0, 0x0)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/credentials_linux.go:172 +0xaac fp=0xc820045850 sp=0xc820045678
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/docker/docker/api/server.(*Server).makeHttpHandler.func1(0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/_build/src/github.com/docker/docker/api/server/server.go:1680 +0x8aa fp=0xc820045aa8 sp=0xc820045850
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.HandlerFunc.ServeHTTP(0xc820180960, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1422 +0x3a fp=0xc820045ac8 sp=0xc820045aa8
marras 12 20:57:43 localhost.localdomain docker[1000]: github.com/gorilla/mux.(*Router).ServeHTTP(0xc8202019a0, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /builddir/build/BUILD/docker-28c300fafb58c380d78381e08e1be35dfed5d4f9/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e fp=0xc820045be0 sp=0xc820045ac8
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.serverHandler.ServeHTTP(0xc820388120, 0x7fdbf128e2e8, 0xc820134580, 0xc8202b0620)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1862 +0x19e fp=0xc820045c40 sp=0xc820045be0
marras 12 20:57:43 localhost.localdomain docker[1000]: net/http.(*conn).serve(0xc8201344d0)
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/net/http/server.go:1361 +0xbee fp=0xc820045f98 sp=0xc820045c40
marras 12 20:57:43 localhost.localdomain docker[1000]: runtime.goexit()
marras 12 20:57:43 localhost.localdomain docker[1000]: /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820045fa0 sp=0xc820045f98
marras 12 20:57:43 localhost.localdomain docker[1000]: created by net/http.(*Server).Serve
@stefwalter

This comment has been minimized.

Show comment
Hide comment
@stefwalter

stefwalter Nov 13, 2015

Contributor

Some more notes:

  • It's great that build output is visible. But it should only be visible with --verbose.
  • Besides space in a qcow2 overlay, is there's any downside to installing in same VM instance?
Contributor

stefwalter commented Nov 13, 2015

Some more notes:

  • It's great that build output is visible. But it should only be visible with --verbose.
  • Besides space in a qcow2 overlay, is there's any downside to installing in same VM instance?
@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 13, 2015

Member

It's great that build output is visible. But it should only be visible with --verbose.

Yes, this was for debugging. I think it can be switched off already.

Alternatively, we can keep the output and make log.html collapse it away by default. What do you think?

Member

mvollmer commented Nov 13, 2015

It's great that build output is visible. But it should only be visible with --verbose.

Yes, this was for debugging. I think it can be switched off already.

Alternatively, we can keep the output and make log.html collapse it away by default. What do you think?

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 13, 2015

Member

Besides space in a qcow2 overlay, is there's any downside to installing in same VM instance?

I don't think so. I'll measure the difference.

Member

mvollmer commented Nov 13, 2015

Besides space in a qcow2 overlay, is there's any downside to installing in same VM instance?

I don't think so. I'll measure the difference.

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 13, 2015

Member

It's great that build output is visible. But it should only be visible with --verbose.

Yes, this was for debugging. I think it can be switched off already.

Heh, I am confused. I added --verbose to the --installdeps step during image creation for debugging. When running testsuite-prepare locally, I don't see any extra output...

Member

mvollmer commented Nov 13, 2015

It's great that build output is visible. But it should only be visible with --verbose.

Yes, this was for debugging. I think it can be switched off already.

Heh, I am confused. I added --verbose to the --installdeps step during image creation for debugging. When running testsuite-prepare locally, I don't see any extra output...

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 13, 2015

Member

It's great that build output is visible. But it should only be visible with --verbose

check-verify passes --verbose to testsuite-prepare by default. I'll switch that off.

Member

mvollmer commented Nov 13, 2015

It's great that build output is visible. But it should only be visible with --verbose

check-verify passes --verbose to testsuite-prepare by default. I'll switch that off.

@stefwalter

This comment has been minimized.

Show comment
Hide comment
@stefwalter

stefwalter Nov 13, 2015

Contributor

Fedora-23 known issues to include in this pull request: stefwalter@4e22388

Contributor

stefwalter commented Nov 13, 2015

Fedora-23 known issues to include in this pull request: stefwalter@4e22388

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 13, 2015

Member

It's great that build output is visible. But it should only be visible with --verbose

check-verify passes --verbose to testsuite-prepare by default. I'll switch that off.

This has been done but it's not effective here since check-verify comes from master during a test run.

Member

mvollmer commented Nov 13, 2015

It's great that build output is visible. But it should only be visible with --verbose

check-verify passes --verbose to testsuite-prepare by default. I'll switch that off.

This has been done but it's not effective here since check-verify comes from master during a test run.

@stefwalter

This comment has been minimized.

Show comment
Hide comment
@stefwalter

stefwalter Nov 13, 2015

Contributor

rhel-7 testSocketPort fix is in #3158

Contributor

stefwalter commented Nov 13, 2015

rhel-7 testSocketPort fix is in #3158

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 13, 2015

Member

#3157 would make f23 pass, but I'll tweak the naughty pattern here as well, just so that it is consistent with the rest.

Member

mvollmer commented Nov 13, 2015

#3157 would make f23 pass, but I'll tweak the naughty pattern here as well, just so that it is consistent with the rest.

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 13, 2015

Member

Rebased, now #3158 is included here.

Member

mvollmer commented Nov 13, 2015

Rebased, now #3158 is included here.

@mvollmer mvollmer removed the blocked label Nov 13, 2015

@stefwalter

This comment has been minimized.

Show comment
Hide comment
@stefwalter

stefwalter Nov 17, 2015

Contributor

Just one last thing. Seen above.

Contributor

stefwalter commented Nov 17, 2015

Just one last thing. Seen above.

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 17, 2015

Member

rhel-7 has storage regressions.

Member

mvollmer commented Nov 17, 2015

rhel-7 has storage regressions.

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 17, 2015

Member

rhel-7 has storage regressions.

Somehow kdump.service makes parted fail. The mdraid failure is #2924, which now seems to happen reliably. Yay!

Member

mvollmer commented Nov 17, 2015

rhel-7 has storage regressions.

Somehow kdump.service makes parted fail. The mdraid failure is #2924, which now seems to happen reliably. Yay!

mvollmer and others added some commits Nov 13, 2015

test: Add known issues for docker panic when serving HTTP
Issue #3156
Reviewed-by: Marius Vollmer <marius.vollmer@redhat.com>
test: Consolidate rhel-7 repositories
Remove unofficial sources for storaged, libssh, and glib2.  We
don't need those anymore.

Remove obsolete hacks re non-functioning repositories.

Add new hack re non-functioning "htb" repository.
@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 17, 2015

Member

I'll investigate the kdump->parted interference a bit more. Disabling kdump probably just changes the timing of another race.

Member

mvollmer commented Nov 17, 2015

I'll investigate the kdump->parted interference a bit more. Disabling kdump probably just changes the timing of another race.

@@ -94,8 +94,6 @@ def install(opts):
cmd = [ "./testsuite-prepare" ]
if opts.clean:
cmd.append("--clean")
if opts.verbosity:
cmd.append("--verbose")

This comment has been minimized.

@dperpeet

dperpeet Nov 18, 2015

Member

why isn't this passed to test-suite-prepare if we want it in check-verify?

@dperpeet

dperpeet Nov 18, 2015

Member

why isn't this passed to test-suite-prepare if we want it in check-verify?

This comment has been minimized.

@mvollmer

mvollmer Nov 18, 2015

Member

This can be debated. check-verify is verbose by default, but testsuite-prepare --verbose is now a lot more verbose than before, so it might be distracting.

@mvollmer

mvollmer Nov 18, 2015

Member

This can be debated. check-verify is verbose by default, but testsuite-prepare --verbose is now a lot more verbose than before, so it might be distracting.

# and will just silently not do it. So we do it here.
#
xfs_growfs /
df -h /

This comment has been minimized.

@dperpeet

dperpeet Nov 18, 2015

Member

This feels like debug output. Do we always want that info?

@dperpeet

dperpeet Nov 18, 2015

Member

This feels like debug output. Do we always want that info?

This comment has been minimized.

@stefwalter

stefwalter Nov 18, 2015

Contributor

I think so. It makes sense to have this when troubleshooting running out of disk space. In fact set -x is also debug output. The creation of a VM gets to be noisy. It isn't done very often.

@stefwalter

stefwalter Nov 18, 2015

Contributor

I think so. It makes sense to have this when troubleshooting running out of disk space. In fact set -x is also debug output. The creation of a VM gets to be noisy. It isn't done very often.

# and will just silently not do it. So we do it here.
#
xfs_growfs /
df -h /

This comment has been minimized.

@dperpeet

dperpeet Nov 18, 2015

Member

same as with fedora 22

@dperpeet

dperpeet Nov 18, 2015

Member

same as with fedora 22

# and will just silently not do it. So we do it here.
#
xfs_growfs /
df -h /

This comment has been minimized.

@dperpeet

dperpeet Nov 18, 2015

Member

same

@dperpeet
yum -y --disablerepo=rhel-7-server-rt-beta-rpms --disablerepo=rhel-sap-hana-for-rhel-7-server-rpms install yum-utils
yum-config-manager --disable rhel-7-server-rt-beta-rpms
yum-config-manager --disable rhel-sap-hana-for-rhel-7-server-rpms
set -x

This comment has been minimized.

@dperpeet

dperpeet Nov 18, 2015

Member

why not set -ex at the beginning like in the other setup scripts?

@dperpeet

dperpeet Nov 18, 2015

Member

why not set -ex at the beginning like in the other setup scripts?

This comment has been minimized.

@mvollmer

mvollmer Nov 18, 2015

Member

"Otherwise the subscription credentials appear in the output above.":)

@mvollmer

mvollmer Nov 18, 2015

Member

"Otherwise the subscription credentials appear in the output above.":)

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 18, 2015

Member

I'll investigate the kdump->parted interference a bit more. Disabling kdump probably just changes the timing of another race.

Got it. Both systemd-udevd and parted are telling the kernel about changed partitions after parted has written the new table, and parted does it in a racy way.

I'll briefly try to figure out what happens on fedora 22/23 and whether this is a recent change in udev etc.

Member

mvollmer commented Nov 18, 2015

I'll investigate the kdump->parted interference a bit more. Disabling kdump probably just changes the timing of another race.

Got it. Both systemd-udevd and parted are telling the kernel about changed partitions after parted has written the new table, and parted does it in a racy way.

I'll briefly try to figure out what happens on fedora 22/23 and whether this is a recent change in udev etc.

@mvollmer

This comment has been minimized.

Show comment
Hide comment
@mvollmer

mvollmer Nov 18, 2015

Member

I'll briefly try to figure out what happens on fedora 22/23 and whether this is a recent change in udev etc.

Fedora 22 doesn't seem to have the issue. See also #3177.

Member

mvollmer commented Nov 18, 2015

I'll briefly try to figure out what happens on fedora 22/23 and whether this is a recent change in udev etc.

Fedora 22 doesn't seem to have the issue. See also #3177.

@mvollmer mvollmer removed the needswork label Nov 18, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment