International characters not supported in codeql-action/analyze ?

[HenrikDK]

Hi

Tried searching docs and issues for this but couldn't find anything.

I'm running the analyze action on a repository with javascript-typescript as the selected language. I'm doing a manual build where i install nodejs 22 before init is called. The job fails with the following error:

Extracting javascript
  /opt/hostedtoolcache/CodeQL/2.20.4/x64/codeql/codeql database trace-command --use-build-mode --working-dir /home/runner/_work/max-optic-pg/max-optic-pg /home/runner/_work/_temp/codeql_databases/javascript
  Running command in /home/runner/_work/max-optic-pg/max-optic-pg: [/opt/hostedtoolcache/CodeQL/2.20.4/x64/codeql/javascript/tools/autobuild.sh]
  [2025-02-20 12:47:44] [build-stdout] Parallel extraction with 8 threads.
  [2025-02-20 12:47:44] [build-stdout] cicd/testing/packages/src/package.json: Main file not found
  [2025-02-20 12:47:44] [build-stdout] src/package.json: Main file set to src/src/index.ts
  [2025-02-20 12:47:44] [build-stdout] Found Node.js at: node
  [2025-02-20 12:47:44] [build-stdout] Found Node.js version: v22.14.0
  [2025-02-20 12:47:44] [build-stdout] Opening project /home/runner/_work/max-optic-pg/max-optic-pg/src/tsconfig.json
  [2025-02-20 12:47:44] [build-stdout] Memory for TypeScript process: 30235 MB, and 400 MB reserve
  [2025-02-20 12:47:45] [build-stdout] Done opening project /home/runner/_work/max-optic-pg/max-optic-pg/src/tsconfig.json (1025 ms)
  [2025-02-20 12:47:46] [build-stderr] java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /home/runner/_work/max-optic-pg/max-optic-pg/src/src/tests/test-data/sk?neTrafikkenOffline.ts
  [2025-02-20 12:47:46] [build-stderr] 	at java.base/sun.nio.fs.UnixPath.encode(Unknown Source)
  [2025-02-20 12:47:46] [build-stderr] 	at java.base/sun.nio.fs.UnixPath.<init>(Unknown Source)
  [2025-02-20 12:47:46] [build-stderr] 	at java.base/sun.nio.fs.UnixFileSystem.getPath(Unknown Source)
  [2025-02-20 12:47:46] [build-stderr] 	at java.base/java.io.File.toPath(Unknown Source)
  [2025-02-20 12:47:46] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.extractTypeScript(AutoBuild.java:1055)
  [2025-02-20 12:47:46] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.extractSource(AutoBuild.java:758)
  [2025-02-20 12:47:46] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.run(AutoBuild.java:464)
  [2025-02-20 12:47:46] [build-stderr] 	at com.semmle.js.extractor.AutoBuild.main(AutoBuild.java:1364)
  Error: 2-20 12:47:46] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/hostedtoolcache/CodeQL/2.20.4/x64/codeql/javascript/tools/autobuild.sh])
  A fatal error occurred: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/2.20.4/x64/codeql/javascript/tools/autobuild.sh]
  Error: Encountered a fatal error while running "/opt/hostedtoolcache/CodeQL/2.20.4/x64/codeql/codeql database trace-command --use-build-mode --working-dir /home/runner/_work/max-optic-pg/max-optic-pg /home/runner/_work/_temp/codeql_databases/javascript". Exit code was 2 and error was: A fatal error occurred: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/2.20.4/x64/codeql/javascript/tools/autobuild.sh]. See the logs for more details.

The offending line is an InvalidPathException:

[2025-02-20 12:47:46] [build-stderr] java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /home/runner/_work/max-optic-pg/max-optic-pg/src/src/tests/test-data/sk?neTrafikkenOffline.ts

The missing character in question is a danish 'å'. But I am somewhat suprised that codeql can't handle international characters in filenames?

Have I missed something? or Is there anyway round this?

This would be a hard block for adoption in our org.

[aibaars]

The java.nio.file package should be able to deal non-ascii characters in file names, however, it will use the system's character encoding to decode file names.

If the LOCALE settings differ from the actual encoding used to create the file on disk then you can get this type of error. What are the LOCALE settings on the Actions runner?

[aibaars]

Could you check that the file name is displayed correctly in GitHub's file browser? If it looks correct, then the file name is likely encoded as UTF-8, which should be the default on the Action runner.

[HenrikDK]

Can confirm that the file name is displayed correctly in githubs file browser:

I will investigate what the local settings are on our internal runners.

[HenrikDK]

Hi again :),

Just confirmed your feedback, locals were not set on our internal runner image. Further more it seemingly isn't set in the oficial runner image that we build on, see docker file here:
https://github.com/actions/runner/blob/main/images/Dockerfile

By adding the following lines to our docker image:

RUN apt-get -y install locales

RUN sudo locale-gen en_US.UTF-8

# set default locales
ENV LANG=en_US.UTF-8 \
    LC_ALL=en_US.UTF-8 \
    LANGUAGE=en_US.UTF-8

Rotating all our runnners, and re-running the codeql job, the job completed succesfully.

Thanks again for the help.

[aibaars]

Great! Thanks for confirming. I created a pull request for the base docker image: actions/runner#3721 . Let's see what the actions team thinks about changing the default locale to UTF-8.