Codeql create golang database error on macos

[p1keman]

codeql-cli v2.7.3
golang v1.17.5
M1 macOS Monterey v12.0.1

I have installed go and added it to the environment variable. I tried adding go environment variables to .zshrc and /etc/paths, but none of them worked.

% go version
go version go1.17.5 darwin/arm64

% codeql database create --language=go test --source-root=../codeql-study --overwrite
Initializing database at /codeql-study/test.
Running build command: [/codeql/cli/codeql/go/tools/autobuild.sh]
[2022-01-02 19:00:46] [build-stderr] 2022/01/02 19:00:46 Unable to run the go command, is it installed?
[2022-01-02 19:00:46] [build-stderr] Error: signal: abort trap
[2022-01-02 19:00:46] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/codeql/cli/codeql/tools/osx64/preload_tracer, /codeql/cli/codeql/go/tools/autobuild.sh])
A fatal error occurred: Exit status 1 from command: [/codeql/cli/codeql/go/tools/autobuild.sh]

[aeisenberg]

Thanks for raising your issue. Mac M1 support is still in beta. Please make sure to read about all of the requirements for running on an M1.

If you have fulfilled all of these requirements and are still seeing the error, please run the command with maximum verbosity and attach the logs created to this issue.

[p1keman]

I have met the requirements mentioned in the document. Codeql can create databases in python and javascript, but errors will occur in go and Java. This is the log of my go database creation.

build-tracer.log

[T 15:28:48 1445] Reading configuration file /Desktop/codeql-study/test/working/tracing/compiler-tracing5901691412012282673.spec...
[T 15:28:48 1445] Compilers file as of 2022-01-03 15:28:48
[T 15:28:48 1445] Compiler pattern: **/go-autobuilder
[T 15:28:48 1445] Replace intercepted call: 0
[T 15:28:48 1445] Extractor: <null>
[T 15:28:48 1445] Compilers file as of 2022-01-03 15:28:48
[T 15:28:48 1445] Compiler pattern: **/go
[T 15:28:48 1445] Replace intercepted call: 0
[T 15:28:48 1445] Extractor: /codeql/cli/codeql/go/tools/osx64/go-extractor
[T 15:28:48 1445] Prepend[0] "--mimic"
[T 15:28:48 1445] Prepend[1] ""${compiler}""
[T 15:28:48 1445] Initialising tags...
[T 15:28:48 1445] ID set to 00000000000005A5_0000000000000001 (parent root)
[T 15:28:48 1445] Reading configuration file /Desktop/codeql-study/test/working/tracing/compiler-tracing5901691412012282673.spec...
[T 15:28:48 1445] Initialising tags...
[T 15:28:48 1445] ID set to 00000000000005A5_0000000000000002 (parent root)
[T 15:28:48 1445] Warning: SEMMLE_EXEC and SEMMLE_EXECP not set. Falling back to path lookup on argv[0].
[T 15:28:48 1445] ==== Candidate to intercept: /codeql/cli/codeql/tools/osx64/runner-osx (canonical: /codeql/cli/codeql/tools/osx64/runner-osx) ====
[T 15:28:48 1446] Attempting to switch stdout/stderr to 3...
/Desktop/codeql-study/test/working/copy-root/000001F5/bin/sh.semmle.000005A5.slice.x86_64: replacing existing signature
[T 15:28:49 1445] Reading configuration file /Desktop/codeql-study/test/working/tracing/compiler-tracing5901691412012282673.spec...
[T 15:28:49 1445] Initialising tags...
[T 15:28:49 1445] ID set to 00000000000005A5_0000000000000003 (parent 00000000000005A5_0000000000000002)
[T 15:28:49 1445] ==== Candidate to intercept: /codeql/cli/codeql/go/tools/autobuild.sh (canonical: /codeql/cli/codeql/go/tools/autobuild.sh) ====
[T 15:28:49 1455] Attempting to switch stdout/stderr to 3...
/Desktop/codeql-study/test/working/copy-root/000001F5/bin/bash.semmle.000005A5.slice.x86_64: replacing existing signature
[T 15:28:49 1445] Reading configuration file /Desktop/codeql-study/test/working/tracing/compiler-tracing5901691412012282673.spec...
[T 15:28:49 1445] Initialising tags...
[T 15:28:49 1445] ID set to 00000000000005A5_0000000000000004 (parent 00000000000005A5_0000000000000003)
[T 15:28:49 1445] ==== Candidate to intercept: /bin/bash (canonical: /bin/bash) ====
[T 15:28:49 1461] Attempting to switch stdout/stderr to 3...
[T 15:28:50 1460] Reading configuration file /Desktop/codeql-study/test/working/tracing/compiler-tracing5901691412012282673.spec...
[T 15:28:50 1460] Initialising tags...
[T 15:28:50 1460] ID set to 00000000000005B4_0000000000000001 (parent 00000000000005A5_0000000000000004)
[T 15:28:50 1460] ==== Candidate to intercept: /codeql/cli/codeql/go/tools/osx64/go-autobuilder (canonical: /codeql/cli/codeql/go/tools/osx64/go-autobuilder) ====
[T 15:28:50 1460] === Intercepted call to /codeql/cli/codeql/go/tools/osx64/go-autobuilder === 
[T 15:28:50 1460] Disabling tracing for this command.
[T 15:28:50 1465] Error: Environment variable SEMMLE_PRELOAD_libtrace is not set (or empty), but it should be set.
[T 15:28:50 1465] Error: Environment variable SEMMLE_PRELOAD_libtrace is not set (or empty), but it should be set.

database-create-20220103.152847.727.log

[2022-01-03 15:28:47] This is codeql database create test --language=go --source-root=./go --overwrite
[2022-01-03 15:28:47] Log file was started late.
[2022-01-03 15:28:47] [PROGRESS] database create> Initializing database at /Desktop/codeql-study/test.
[2022-01-03 15:28:47] Running plumbing command: codeql database init --overwrite --language=go --source-root=/Desktop/codeql-study/go --allow-missing-source-root=false --allow-already-existing -- /Desktop/codeql-study/test
[2022-01-03 15:28:47] [PROGRESS] database init> Counting lines of code in /Desktop/codeql-study/go
[2022-01-03 15:28:47] [PROGRESS] database init> Resolving extractor go.
[2022-01-03 15:28:47] Calling plumbing command: codeql resolve languages --format=json
[2022-01-03 15:28:47] [DETAILS] resolve languages> Scanning for [codeql-extractor.yml] from /codeql/cli/codeql/.codeqlmanifest.json
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/go/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/python/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/java/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/html/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/xml/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/properties/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/cpp/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/csv/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/csharp/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/javascript/codeql-extractor.yml.
[2022-01-03 15:28:47] [DETAILS] resolve languages> Parsing /codeql/cli/codeql/ruby/codeql-extractor.yml.
[2022-01-03 15:28:48] Plumbing command codeql resolve languages completed:
                      {
                        "extractors" : {
                          "go" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/go"
                            }
                          ],
                          "python" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/python"
                            }
                          ],
                          "java" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/java"
                            }
                          ],
                          "html" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/html"
                            }
                          ],
                          "xml" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/xml"
                            }
                          ],
                          "properties" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/properties"
                            }
                          ],
                          "cpp" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/cpp"
                            }
                          ],
                          "csv" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/csv"
                            }
                          ],
                          "csharp" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/csharp"
                            }
                          ],
                          "javascript" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/javascript"
                            }
                          ],
                          "ruby" : [
                            {
                              "extractor_root" : "/codeql/cli/codeql/ruby"
                            }
                          ]
                        }
                      }
[2022-01-03 15:28:48] [DETAILS] database init> Found candidate extractor root for go: /codeql/cli/codeql/go.
[2022-01-03 15:28:48] [PROGRESS] database init> Successfully loaded extractor Go (go) from /codeql/cli/codeql/go.
[2022-01-03 15:28:48] [PROGRESS] database init> Created skeleton CodeQL database at /Desktop/codeql-study/test. This in-progress database is ready to be populated by an extractor.
[2022-01-03 15:28:48] Plumbing command codeql database init completed.
[2022-01-03 15:28:48] Using autobuild script /codeql/cli/codeql/go/tools/autobuild.sh.
[2022-01-03 15:28:48] [DETAILS] database create> Detected autobuild command for Go: /codeql/cli/codeql/go/tools/autobuild.sh.
[2022-01-03 15:28:48] [PROGRESS] database create> Running build command: [/codeql/cli/codeql/go/tools/autobuild.sh]
[2022-01-03 15:28:48] Running plumbing command: codeql database trace-command --working-dir=/Desktop/codeql-study/go --index-traceless-dbs --no-db-cluster -- /Desktop/codeql-study/test /codeql/cli/codeql/go/tools/autobuild.sh
[2022-01-03 15:28:48] Setting up tracing from specification at /codeql/cli/codeql/go/tools/osx64/compiler-tracing.spec.
[2022-01-03 15:28:48] [PROGRESS] database trace-command> Running command in /Desktop/codeql-study/go: [/codeql/cli/codeql/go/tools/autobuild.sh]
[2022-01-03 15:28:50] [build-stderr] 2022/01/03 15:28:50 Unable to run the go command, is it installed?
[2022-01-03 15:28:50] [build-stderr] Error: signal: abort trap
[2022-01-03 15:28:50] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/codeql/cli/codeql/tools/osx64/preload_tracer, /codeql/cli/codeql/go/tools/autobuild.sh])
[2022-01-03 15:28:50] Exception caught at top level: Exit status 1 from command: [/codeql/cli/codeql/go/tools/autobuild.sh]
                      com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:188)
                      com.semmle.cli2.database.TraceCommandCommand.executeSubcommand(TraceCommandCommand.java:86)
                      com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
                      com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:162)
                      com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:151)
                      com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:500)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:205)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:214)
                      com.semmle.cli2.CodeQL.main(CodeQL.java:98)

[edoardopirovano]

Hmm, there is a known issue with tracing Java builds on M1 that will be fixed in the next release: github/codeql-cli-binaries#101

Looking at that log, this appears to be a separate issue however. @AlexDenisov do you agree?

[AlexDenisov]

@edoardopirovano it manifests in a different way, but the root cause is the same.

@p1keman at this point I can recommend either using CodeQL 2.7.1, or waiting for the next release (which is around the corner, but no exact ETA yet).

[p1keman]

@edoardopirovano it manifests in a different way, but the root cause is the same.

@p1keman at this point I can recommend either using CodeQL 2.7.1, or waiting for the next release (which is around the corner, but no exact ETA yet).

Very good, I switched to codeql v2.7.1, can create a go database, thank you