Use GHC_CHARENC=UTF-8 when calling GHC

[nad]

We use hGetContents in callCompiler':

agda/src/full/Agda/Compiler/CallCompiler.hs

Lines 55 to 76 in 9399b3a

	(_, out, err, p) <-
	liftIO $ createProcess
	(proc cmd args) { std_err = CreatePipe
	, std_out = CreatePipe
	}
	-- In -v0 mode we throw away any progress information printed to
	-- stdout.
	case out of
	Nothing -> __IMPOSSIBLE__
	Just out -> forkTCM $ do
	-- The handle should be in text mode.
	liftIO $ hSetBinaryMode out False
	progressInfo <- liftIO $ hGetContents out
	mapM_ (reportSLn "compile.output" 1) $ lines progressInfo
	errors <- liftIO $ case err of
	Nothing -> __IMPOSSIBLE__
	Just err -> do
	-- The handle should be in text mode.
	hSetBinaryMode err False
	hGetContents err

The handles out and err use the default (locale) encoding. That is perhaps appropriate, if we know nothing about the encoding used by the process that we call. We only use callCompiler' to call GHC. Does GHC use the locale encoding when writing to stdout or stderr?

[nad]

If the environment variable GHC_CHARENC is set to UTF-8, then GHC appears to use UTF-8 for stdout and stderr:

https://github.com/ghc/ghc/blob/0dc723957d0fdb5909f145405b775efea0fe2f6e/ghc/Main.hs#L113-L118

https://github.com/ghc/ghc/blob/0dc723957d0fdb5909f145405b775efea0fe2f6e/libraries/ghc-boot/GHC/HandleEncoding.hs#L9-L17

The variable GHC_CHARENC does not seem to be mentioned in the GHC documentation, but I still think we should make use of it. It seems to be available from at least GHC 8.

[nad]

Perhaps callCompiler or callCompiler' are used by other packages. I suggest that we add an optional encoding argument to these procedures.

[andreasabel]

The variable GHC_CHARENC does not seem to be mentioned in the GHC documentation,

I reported this upstream: https://gitlab.haskell.org/ghc/ghc/-/issues/20963