Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration test failures #2

Closed
rosenhouse opened this issue Dec 8, 2015 · 5 comments
Closed

Integration test failures #2

rosenhouse opened this issue Dec 8, 2015 · 5 comments
Labels
unscheduled

Comments

@rosenhouse
Copy link
Contributor

Hi all,

We're running the test suite again, this time getting a different error:

scripts/remote-fly ci/nested-tests.yml
executing build 1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21.6M    0 21.6M    0     0  12.2M      0 --:--:--  0:00:01 --:--:-- 12.2M
initializing with docker:///cloudfoundry/garden-ci-ubuntu
running guardian-release/ci/scripts/nested-tests
++ dirname guardian-release/ci/scripts/nested-tests
+ cd guardian-release/ci/scripts/../..
+ export GOROOT=/usr/local/go
+ GOROOT=/usr/local/go
+ export PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export GOPATH=/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release
+ GOPATH=/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release
+ export PATH=/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ pushd src/github.com/cloudfoundry-incubator/guardian
/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/guardian /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release
+ go install -tags daemon github.com/cloudfoundry-incubator/guardian/cmd/guardian/
+ go install github.com/cloudfoundry-incubator/guardian/rundmc/iodaemon/cmd/iodaemon
+ tmpdir=/tmp/dir
+ rm -fr /tmp/dir
+ mkdir -p /tmp/dir/depot
+ mkdir /tmp/dir/snapshots
+ mkdir /tmp/dir/graph
+ popd
/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release
+ go install github.com/onsi/ginkgo/ginkgo
+ /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/bin/guardian -iodaemonBin=/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/bin/iodaemon -depot=/tmp/dir/depot -snapshots=/tmp/dir/snapshots -graph=/tmp/dir/graph -bin=/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/guardian/linux_backend/bin -listenNetwork=tcp -listenAddr=0.0.0.0:7777 -denyNetworks= -allowNetworks= -allowHostAccess=false -mtu=1500 -containerGraceTime=5m -logLevel=error -rootfs=/opt/warden/rootfs
ERRO[0000] Failed to GetDriver graph btrfs /tmp/dir/graph
ERRO[0000] Failed to GetDriver graph zfs /tmp/dir/graph
ERRO[0000] Failed to GetDriver graph devicemapper /tmp/dir/graph
ERRO[0000] Failed to GetDriver graph overlay /tmp/dir/graph
ERRO[0000] Failed to GetDriver graph vfs /tmp/dir/graph
{"timestamp":"1449606737.425148010","source":"guardian","message":"guardian.volume-creator.failed-to-construct-graph-driver","log_level":3,"data":{"error":"No supported storage backend found","graphRoot":"/tmp/dir/graph","session":"6","trace":"goroutine 1 [running]:\ngithub.com/pivotal-golang/lager.(*logger).Fatal(0xc2080785a0, 0xa41190, 0x20, 0x7f5eccc41e80, 0xc208095d50, 0x0, 0x0, 0x0)\n\t/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/pivotal-golang/lager/logger.go:131 +0xc8\nmain.wireVolumeCreator(0x7f5eccc46e40, 0xc2080785a0, 0x7ffc7a3b3c47, 0xe, 0xc20806db00)\n\t/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/guardian/cmd/guardian/main.go:343 +0x3fe\nmain.main()\n\t/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/guardian/cmd/guardian/main.go:236 +0x5c4\n"}}
panic: No supported storage backend found

goroutine 1 [running]:
github.com/pivotal-golang/lager.(*logger).Fatal(0xc2080785a0, 0xa41190, 0x20, 0x7f5eccc41e80, 0xc208095d50, 0x0, 0x0, 0x0)
    /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/pivotal-golang/lager/logger.go:152 +0x5d0
main.wireVolumeCreator(0x7f5eccc46e40, 0xc2080785a0, 0x7ffc7a3b3c47, 0xe, 0xc20806db00)
    /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/guardian/cmd/guardian/main.go:343 +0x3fe
main.main()
    /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/guardian/cmd/guardian/main.go:236 +0x5c4

goroutine 5 [syscall]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/local/go/src/os/signal/signal_unix.go:27 +0x35
+ cd src/github.com/cloudfoundry-incubator/garden-integration-tests
++ hostname
+ export GARDEN_ADDRESS=7g89flel121:7777
+ GARDEN_ADDRESS=7g89flel121:7777
+ ginkgo -p -nodes=4
Running Suite: GardenIntegrationTests Suite
===========================================
Random Seed: 1449606738
Will run 112 of 113 specs

Running in parallel across 4 nodes

• Failure in Spec Setup (JustBeforeEach) [0.003 seconds]
Limits [JustBeforeEach]
/tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/garden-integration-tests/limits_test.go:346
  LimitMemory
  /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/garden-integration-tests/limits_test.go:33
    with a memory limit
    /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/garden-integration-tests/limits_test.go:32
      when the process writes too much to /dev/shm
      /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/garden-integration-tests/limits_test.go:31
        is killed
        /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/garden-integration-tests/limits_test.go:30

        Expected error:
            <*url.Error | 0xc2080b9f50>: {
                Op: "Post",
                URL: "http://api/containers",
                Err: {
                    Op: "dial",
                    Net: "tcp",
                    Addr: {
                        IP: "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\n\xfe\x00\x06",
                        Port: 7777,
                        Zone: "",
                    },
                    Err: 0x6f,
                },
            }
            Post http://api/containers: dial tcp 10.254.0.6:7777: connection refused
        not to have occurred

        /tmp/build/ee32fe39-4383-4d2d-50d8-76e8ae42e05a/guardian-release/src/github.com/cloudfoundry-incubator/garden-integration-tests/garden_integration_tests_suite_test.go:52

and more failures like this afterwards.

We saw the error on both a local Concourse lite and also on an AWS concourse worker.

Have we misconfigured something?

Thanks!

cc @dbellotti
@cf-gitbot
Copy link

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

@julz
Copy link
Contributor

julz commented Dec 8, 2015

If you're not using the latest concourse - and obviously that's the recommended solution - you'll need to log in to the workers (or vagrant ssh in concourse-lite) and run 'modprobe aufs'. This is unfortunately necessary to be able to run guardian inside a container because it relies on aufs support which can only be enabled in the host. In the latest concourse/garden combination this happens automatically for you.

@rosenhouse
Copy link
Contributor Author

We are running:

  • concourse/0.68.0
  • garden-linux/0.328.0

This is the latest combination listed on the Concourse releases page.

Should we upgrade the version of garden-linux beyond 0.328?

@dbellotti
Copy link

@julz we bosh ssh'd onto the worker VM and ran lsmod:

lsmod
Module                  Size  Used by
dummy                  16384  0
veth                   16384  0
ipt_MASQUERADE         16384  14
nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
bridge                110592  0
stp                    16384  1 bridge
nf_log_ipv4            16384  1
llc                    16384  2 stp,bridge
nf_log_common          16384  1 nf_log_ipv4
xt_LOG                 16384  1
iptable_nat            16384  1
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 24576  2 nf_nat_ipv4,nf_nat_masquerade_ipv4
ipt_REJECT             16384  1
nf_reject_ipv4         16384  1 ipt_REJECT
nf_conntrack_ipv4      16384  4
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
xt_conntrack           16384  3
nf_conntrack          106496  5 nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4
iptable_filter         16384  1
ip_tables              28672  2 iptable_filter,iptable_nat
x_tables               36864  6 ip_tables,ipt_MASQUERADE,xt_conntrack,xt_LOG,iptable_filter,ipt_REJECT
aufs                  212992  1
btrfs                 937984  1
xor                    24576  1 btrfs
raid6_pq               98304  1 btrfs
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
ghash_clmulni_intel    16384  0
aesni_intel           172032  0
cirrus                 28672  1
aes_x86_64             20480  1 aesni_intel
lrw                    16384  1 aesni_intel
gf128mul               16384  1 lrw
ppdev                  20480  0
ttm                    94208  1 cirrus
glue_helper            16384  1 aesni_intel
drm_kms_helper        126976  1 cirrus
drm                   344064  4 ttm,drm_kms_helper,cirrus
xen_fbfront            20480  0
ablk_helper            16384  1 aesni_intel
fb_sys_fops            16384  1 xen_fbfront
cryptd                 20480  3 ghash_clmulni_intel,aesni_intel,ablk_helper
syscopyarea            16384  2 cirrus,xen_fbfront
sysfillrect            16384  2 cirrus,xen_fbfront
sysimgblt              16384  2 cirrus,xen_fbfront
i2c_piix4              24576  0
8250_fintek            16384  0
serio_raw              16384  0
mac_hid                16384  0
parport_pc             32768  0
parport                45056  2 ppdev,parport_pc
nfsd                  294912  2
auth_rpcgss            61440  1 nfsd
nfs_acl                16384  1 nfsd
nfs                   245760  0
lockd                  94208  2 nfs,nfsd
grace                  16384  2 nfsd,lockd
sunrpc                327680  6 nfs,nfsd,auth_rpcgss,lockd,nfs_acl
fscache                65536  1 nfs
psmouse               114688  0
floppy                 77824  0
pata_acpi              16384  0

Just re-ran, same error.

@rosenhouse
Copy link
Contributor Author

Just to follow-up here based on a conversation in slack, this test script is not currently part of the "supported" test suite -- it is legacy from garden-linux and hasn't been fully ported over yet to guardian. So I'm closing this issue.

totherme added a commit that referenced this issue Jan 12, 2016
@totherme @julz
bump garden-shed
720b1b8 
[finishes #111305206]

Submodule src/github.com/cloudfoundry-incubator/garden-shed c0694b3..dfb0718:
  > Make test more deterministic
  > Mount loop devices with journalling off, writeback, noatime
  > Handle official images only when using DockerHub
  > Improve logging of graph clean up algorithm
  > Merge pull request #2 from kdykeman/master
  > Add prefix to temp files

Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>
teddyking pushed a commit that referenced this issue May 3, 2017
@teddyking
bump garden-shed guardian idmapper
926c823 
[#144028139]

Submodule src/code.cloudfoundry.org/garden-shed 3c13c8d..551b6de:
  > Move rootfs_provider.MappingList to idmapper
Submodule src/code.cloudfoundry.org/guardian cf6012a..95449df:
  > Parameterise runc root
  > Separate UID and GID mappings
  > Calculate ID mapping via idmapper.MappingsForUser
Submodule src/code.cloudfoundry.org/idmapper 5596f32..e95cf67:
  > Adds glide config
  > Merge pull request #2 from cloudfoundry/mappings-for-user
  > Add Readme

Signed-off-by: Ed King <eking@pivotal.io>
BooleanCat pushed a commit that referenced this issue Mar 27, 2019
bump netplugin-shim
02a390b 
Submodule src/netplugin-shim b18f0ca..9cf811f:
  > Merge pull request #2 from christopherclark/master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
unscheduled
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dbellotti @julz @rosenhouse @cf-gitbot