gcc suddenly disappeared from %PATH%

[mholt]

Our CI builds have been going along their merry way, until recently they all started failing with some unrelated error about gcc not being in the %PATH%.

Here is the first build where it started: https://ci.appveyor.com/project/mholt/caddy/build/4053

The log says:

exec: "gcc": executable file not found in %PATH%

(This is the output from go test -race which invokes cgo, which of course uses gcc to compile the C code.)

Our appveyor.yml file is:

version: "{build}"

hosts:
  quic.clemente.io: 127.0.0.1

os: Windows Server 2012 R2

clone_folder: c:\gopath\src\github.com\mholt\caddy

environment:
  GOPATH: c:\gopath

install:
  - rmdir c:\go /s /q
  - appveyor DownloadFile https://storage.googleapis.com/golang/go1.11.windows-amd64.zip
  - 7z x go1.11.windows-amd64.zip -y -oC:\ > NUL
  - set PATH=%GOPATH%\bin;%PATH%
  - go version
  - go env
  - go get -t ./...
  - go get github.com/golang/lint/golint
  - go get github.com/FiloSottile/vendorcheck
  # Install gometalinter
  - go get github.com/alecthomas/gometalinter

build: off

test_script:
  - gometalinter --install
  - gometalinter --disable-all -E vet -E gofmt -E misspell -E ineffassign -E goimports -E deadcode --tests --vendor ./...
  - vendorcheck ./...
  - go test -race ./...

after_test:
  - golint ./...

deploy: off

And has been unchanged for a while, except for one commit, which merely bumps the version of Go:

diff --git a/appveyor.yml b/appveyor.yml
index 24f5d98..4a79f83 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -12,8 +12,8 @@ environment:
 
 install:
   - rmdir c:\go /s /q
-  - appveyor DownloadFile https://storage.googleapis.com/golang/go1.10.windows-amd64.zip
-  - 7z x go1.10.windows-amd64.zip -y -oC:\ > NUL
+  - appveyor DownloadFile https://storage.googleapis.com/golang/go1.11.windows-amd64.zip
+  - 7z x go1.11.windows-amd64.zip -y -oC:\ > NUL
   - set PATH=%GOPATH%\bin;%PATH%
   - go version
   - go env

Although that change and the PATH error are similar in time, it's unclear why or if that change would cause it. Hence I'm asking for help. Nobody else seems to be having this problem, and I can't find it by googling. Any ideas how to fix our builds?

[Wasapl]

This might be caused by recent image update https://www.appveyor.com/updates/2018/09/01/.
Can you please add next line in appveyor.yml and test it:

image: Previous Visual Studio 2017

or

image: Previous Visual Studio 2015

[mholt]

Thanks for the tip. Unfortunately, neither image worked. (See linked issue for CI builds.) Any other ideas? :)

[Wasapl]

As I can see there is several gcc.exe provided by different tools in appveyor's Image:

C:\MinGW\bin
C:\msys64\mingw32\bin
C:\msys64\mingw64\bin
C:\msys64\usr\bin
C:\Qt\Tools\mingw492_32\bin
C:\Qt\Tools\mingw530_32\bin
C:\Ruby23\DevKit\mingw\bin
C:\Ruby23-x64\DevKit\mingw\bin
C:\Ruby193\DevKit\mingw\bin
C:\cygwin64\bin

You can manually add appropriate path from the list above to %PATH%.
You can test gcc is available with next command:
where gcc.exe

For example:

C:>set PATH=c:\MinGW\bin;%PATH%

C:\>where gcc.exe
c:\MinGW\bin\gcc.exe

C:\>

[mholt]

Thanks. Any idea what the default %PATH% is? I guess it's not in any of those.

I still can't figure out why it seemed to change/break suddenly. Any way I can run that where command interactively?

[Wasapl]

I can't figure it out either.
I do confirm that gcc.exe not in %PATH% in Previous Visual Studio 2015 image.

You can RDP to build worker and run where command interactively: https://www.appveyor.com/docs/how-to/rdp-to-build-worker/

[Wasapl]

I just tested that next line makes gcc available:
set PATH=C:\msys64\mingw64\bin;%PATH%

I lean to the idea that one of the go modules you use started to require gcc recently.

BTW, may I suggest you to use go1.11 which preinstalled in our images?
You can remove lines which download and upzip go1.11 from appveyo.yml and add:
stage: go 1.11
More info about stacks is here https://www.appveyor.com/docs/lang/go/

[mholt]

Oh, awesome! Is that stack new? I don't remember seeing it a few years ago when I set up my appveyor script.

(go test -race, which runs go test with the race detector, has always worked fine for the last few years. I'm surprised that it "suddenly" requires cgo -- in fact, I'm pretty sure it always has.)

I will try this next chance I get and report back!

[Wasapl]

Well, Appveyor constantly evolve 😄 No wonder there is a lot of changes since your first set up.

We started to support Linux recently, BTW. You can configure one appveyor.yml for windows and Ubuntu builds. More info here.

[mholt]

@Wasapl Thanks for the tip about the stack; I'm using that now, but unfortunately the error remains: https://ci.appveyor.com/project/mholt/caddy/build/4077

Now trying to manually set the PATH so that gcc is available as you suggested above...

[mholt]

Adding gcc to the PATH manually fixed it.

Strange. I know cgo was used before since we always used go test -race. Thanks for the help!