Unicode Symbols replaced with ? on Windows

[scottmckendry]

Neovim version (nvim -v)

v0.10.0-dev-2098+g92672a161

Operating system/version

Windows 11 23H2

Add the debug logs

• I have set log_level = vim.log.levels.DEBUG and pasted the log contents below.

Log file

15:54:22[DEBUG] Running formatters on C:\Users\scott\git\windots\test.ps1: { "powershell" }
15:54:22[INFO] Run powershell on C:\Users\scott\git\windots\test.ps1
15:54:22[TRACE] Input lines: { '$emoji = "👍"', '$folderIconNerdFont = ""' }
15:54:22[DEBUG] Run command: { "pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", "(Invoke-Formatter", "(Get-Content -Raw -Path", "C:\Users\scott\git\windots\test.ps1", ")).Trim()" }
15:54:23[DEBUG] powershell exited with code 0
15:54:23[TRACE] Output lines: { '$emoji = "??"\r', '$folderIconNerdFont = "?"\r' }
15:54:23[TRACE] powershell stderr: { "" }
15:54:23[TRACE] Applying formatting to C:\Users\scott\git\windots\test.ps1
15:54:23[TRACE] Comparing lines { '$emoji = "👍"', '$folderIconNerdFont = ""' } and { '$emoji = "??"\r', '$folderIconNerdFont = "?"\r' }
15:54:23[TRACE] Diff indices { { 1, 2, 1, 2 } }
15:54:23[TRACE] Applying text edits: { {
newText = '??"\r\n$folderIconNerdFont = "?"\r',
range = {
["end"] = {
character = 27,
line = 1
},
start = {
character = 10,
line = 0
}
}
} }
15:54:23[TRACE] Done formatting C:\Users\scott\git\windots\test.ps1

Describe the bug

This is isolated to Windows only. I've tested this on Linux with the same config and format command and it works perfectly.

Unicode chars like emoji and icons get replaced with ?. At present, I can only replicate this with PowerShell but wouldn't be surprised if other formatters are affected.

Before:

After:

Running the format command directly from the command line works fine. So I can only assume there is something weird about how Conform is handling the output.

What is the severity of this bug?

breaking (some functionality is broken)

Steps To Reproduce

1. nvim -u minimal.lua
2. :e test.ps1
3. enter emojis, nerd font icons etc.
4. save

Expected Behavior

Characters should not change during formatting.

Minimal example file

$emoji = "👍"
$folderIconNerdFont = ""

Minimal init.lua

-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
    vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
    vim.fn.system({
        "git",
        "clone",
        "--filter=blob:none",
        "--single-branch",
        "https://github.com/folke/lazy.nvim.git",
        lazypath,
    })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
    "folke/tokyonight.nvim",
    {
        "stevearc/conform.nvim",
        config = function()
            require("conform").setup({
                log_level = vim.log.levels.DEBUG,
                -- add your config here
                formatters_by_ft = {
                    ps1 = { "powershell" },
                },

                format_after_save = {
                    lsp_fallback = true,
                },

                formatters = {
                    powershell = {
                        command = "pwsh",
                        args = {
                            "-NoLogo",
                            "-NoProfile",
                            "-NonInteractive",
                            "-Command",
                            "(Invoke-Formatter",
                            "(Get-Content -Raw -Path",
                            "$FILENAME",
                            ")).Trim()",
                        },
                    },
                },
            })
        end,
    },
    -- add any other plugins here
}
require("lazy").setup(plugins, {
    root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

Additional context

No response

[scottmckendry]

For some clarity, when testing on formatting cmd on its own, I'm running pwsh -NoProfile -NoLogo -Command "(Invoke-Formatter (Get-Content -Raw -Path ./test.ps1)).Trim()"

Which writes the file to stdout as expected:

[c00t]

conform.nvim could be calling pwsh from cmd? And then reading stdout? So your script example doesn't prove anything (pwsh can handle unicode well). This is most likely a cmd problem?

I just solved a encoding problem on windows due to a cmd AutoRun script...

code ref:

conform.nvim/lua/conform/runner.lua

Lines 322 to 334 in 192a6d2

	local ok, jid_or_err = pcall(vim.fn.jobstart, cmd, {
	cwd = cwd,
	env = env,
	stdout_buffered = true,
	stderr_buffered = true,
	stdin = config.stdin and "pipe" or "null",
	on_stdout = function(_, data)
	if config.stdin then
	stdout = data
	end
	end,
	on_stderr = function(_, data)
	stderr = data

you can just log data in on_stdout to track stdout by your command.

---

emm, I believe vim.fn.jobstart uses platform specific shell .

just thought of another possibility, vim.fn.jobstart use a different profile(so different shell encoding) than the one used for manually calling pwsh