Tests are failing on gotip

[williammartin]

• Failure [0.002 seconds]
MatchErrorMatcher [It] shows negated failure message
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:111

  Expected
      <string>: Expected
          <*errors.errorString | 0xc0002ec750>: {
              s: "foo",
              frame: {
                  frames: [0x10a23b1, 0x164115e, 0x15e5f58],
              },
          }
      not to match error
          <string>: foo
  to contain substring
      <string>: {s: "foo"}
      not to match error
          <string>: foo

  /Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:115
------------------------------
• Failure [0.002 seconds]
MatchErrorMatcher [It] shows failure message
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:104

  Expected
      <string>: Expected
          <*errors.errorString | 0xc000324c00>: {
              s: "foo",
              frame: {
                  frames: [0x10a23b1, 0x1640bde, 0x15e5f58],
              },
          }
      to match error
          <string>: bar
  to contain substring
      <string>: {s: "foo"}
      to match error
          <string>: bar

  /Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:108
• Failure [0.002 seconds]
MatchErrorMatcher When asserting against an error [It] should succeed when matching with an error
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:21

  Expected
      <*errors.errorString | 0xc0001c7680>: {
          s: "an error",
          frame: {
              frames: [0x10a23b1, 0x163dd8c, 0x15c84fe],
          },
      }
  to match error
      <*errors.errorString | 0xc0001c76b0>: {
          s: "an error",
          frame: {
              frames: [0x10a23b1, 0x163df22, 0x15c84fe],
          },
      }

  /Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:26
------------------------------
• Failure [0.002 seconds]
Equal When asserting equality between objects [It] should do the right thing
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/equal_matcher_test.go:23

  Expected
      <*errors.errorString | 0xc0003fef90>: {
          s: "foo",
          frame: {
              frames: [0x10a23b1, 0x1629770, 0x15c84fe],
          },
      }
  to equal
      <*errors.errorString | 0xc0003fefc0>: {
          s: "foo",
          frame: {
              frames: [0x10a23b1, 0x16298ca, 0x15c84fe],
          },
      }

  /Users/pivotal/go/src/github.com/onsi/gomega/matchers/equal_matcher_test.go:38
------------------------------
• Failure [0.002 seconds]
Succeed [It] builds failure message
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/succeed_matcher_test.go:62

  Expected
      <string>: "...0>: {
           ..."
  to equal               |
      <string>: "...0>: {s: "oo..."

  /Users/pivotal/go/src/github.com/onsi/gomega/matchers/succeed_matcher_test.go:65

Summarizing 5 Failures:

[Fail] MatchErrorMatcher [It] shows negated failure message
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:115

[Fail] MatchErrorMatcher [It] shows failure message
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:108

[Fail] MatchErrorMatcher When asserting against an error [It] should succeed when matching with an error
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/match_error_matcher_test.go:26

[Fail] Equal When asserting equality between objects [It] should do the right thing
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/equal_matcher_test.go:38

[Fail] Succeed [It] builds failure message
/Users/pivotal/go/src/github.com/onsi/gomega/matchers/succeed_matcher_test.go:65

[williammartin]

Git bisect strikes again! This is the offending git commit:

golang/go@37f8481

[williammartin]

I suspect that:

golang/go@b9596ae
golang/go@b34c5f0
golang/go@36b09f3

Are all to prepare for this exact breakage.

[williammartin]

FWIW, this is the script I used to bisect go:

#!/bin/bash

set -x

pushd src
  ./all.bash
popd

pushd "$GOPATH/src/github.com/onsi/gomega"
	ginkgo -p -r --randomizeAllSpecs --failOnPending --randomizeSuites --race matchers
  exit_code=$?

  if [[ "$exit_code" -eq "197" ]]; then
    exit_code=0
  fi
popd

exit $exit_code

I also commented out https://github.com/golang/go/blob/master/src/all.bash#L13 just to speed it up. From the go git repository I then ran:

export PATH=<PATH_TO_WHERE_GO_BUILDS>:$PATH
git bisect start
git bisect good 05e77d4191
git bisect bad 4e2b0dda8c
git bisect run <path-to-script>

In my case, the path where go builds was /Users/pivotal/workspace/go/bin, so I guess it is $GOPATH/bin.

Additionally, I focused the particular tests that were failing in the matcher suite to speed up the bisect, which is why I added:

if [[ "$exit_code" -eq "197" ]]; then
  exit_code=0
fi

Ginkgo exits 197 if you have a programmatic focus set.

This took maybe 5-10 minutes. Go and get a coffee.

[williammartin]

I left a comment about this on the errors proposal in Go: golang/go#29934 (comment)

[jba]

The fix is to use something other than reflect.DeepEqual to compare errors. Perhaps compare the error string?

[williammartin]

@jba Thanks, that's what I was proposing in #333, though as I (under the guise of my colleague teddyking, whoops) commented, type information is important to capture as well. I think we can probably still do that.

[williammartin]

I think I'm going to stop testing against gotip because the failures are causing confusion. Then I'm going to create a branch that is testing against gotip into which I can fix these changes without pressure :)

[IngCr3at1on]

scratch that... for some reason I didn't see the accompanying PR where you were already looking at the included matcher... ignore me lol

[williammartin]

Looks like reflect.DeepEqual will be changed to ignore frames so we won't need any changes golang/go#29934 (comment)