UTF-8/Glyphs not working in macOS

[R-7200]

Hi!

I compiled vis-0.7 via pkgsrc/wip on macOS 12.6.4 (21G526), using iTerm2.app Build 3.5.0beta10. Compilation of vis worked :-) Now the following problem:

mk ~ $ cat utf8-test.txt
äüöß
ÄÖÜß

mk ~ $ file utf8-test.txt
utf8-test.txt: Unicode text, UTF-8 text

mk ~ $ locale
LANG="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_CTYPE="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_ALL="de_DE.UTF-8"

mk ~/pkgsrc/wip (master) $ vis-editor -v
vis v0.7 +curses +lua

Result is as following:

Tested with Terminal.app, XQuartz.app and different $LANG, $LC_ALL. Vis in OpenBSD 7.3 (via ssh/mosh) is showing the Umlaute without problems. Same locale-settings. Both systems have a running tmux session, from where vis is started.

Looking forward to suggestions...

Kind regards.

[erf]

I've set the following:

export LC_ALL=en_US.UTF-8
export LC_COLLATE=C

and tested your input works ok

[R-7200]

Hi!

Tested with your suggestions:

No luck :-(

[rnpnr]

I don't have access to macOS to test anything but can you confirm if
this is an issue with a default visrc.lua? I see you have at least
one plugin loaded.

Also v0.7 is quite old. I briefly looked over the commits since then
and I don't think anything changed that would effect that but can you
confirm if its an issue with the current version?

v0.7...master

[R-7200]

Hi!

Disabled visrc.lua and updated to vis 0.8:

[rnpnr]

And this is running through tmux? If so is it still an issue when run
directly without tmux in the middle?

[erf]

Can you print the configureoutput? Maybe you're using ncurses and not ncursesw to support wide characters? Probably not the issue, but maybe check your terminal font

[R-7200]

I feel stupid now :-( pkgsrc/wip port is using curses:

(1) mk ~/.../wip/vis-editor (master !?) $ sudo bmake NO_CHECKSUM=yes
=> Bootstrap dependency digest>=20211023: found digest-20220214
===> Installing dependencies for vis-editor-0.8
==========================================================================
The following variables will affect the build process of this package,
vis-editor-0.8.  Their current value is shown below:

        * CURSES_DEFAULT = ncurses
        * LUA_VERSION_DEFAULT = 54
        * READLINE_DEFAULT = editline

Based on these variables, the following variables have been set:

        * CURSES_TYPE = ncurses
        * LUA_PACKAGE = lua54
        * READLINE_TYPE = editline
        * TERMCAP_TYPE = curses

You may want to abort the process now with CTRL-C and change the value
of variables in the first group before continuing.  Be sure to run
`/opt/pkg/bin/bmake clean' after the changes.
==========================================================================
=> Tool dependency mktools-[0-9]*: found mktools-20220614
=> Tool dependency pkgconf-[0-9]*: found pkgconf-1.9.4nb1
=> Tool dependency cwrappers>=20150314: found cwrappers-20220403
=> Tool dependency checkperms>=1.1: found checkperms-1.12
=> Full dependency lua54-lpeg-[0-9]*: found lua54-lpeg-1.0.2
=> Full dependency lua54>=5.4.0<5.5: found lua54-5.4.6
=> Full dependency ncursesw>=6.0: found ncursesw-6.4
=> Full dependency libtermkey>=0.17: found libtermkey-0.22
===> Checking for vulnerabilities in vis-editor-0.8
===> Overriding tools for vis-editor-0.8
===> Extracting for vis-editor-0.8
===> Patching for vis-editor-0.8
===> Creating toolchain wrappers for vis-editor-0.8
/bin/ln -s /Users/mk/pkgsrc/wip/vis-editor/work/.buildlink/lib/pkgconfig/lua-5.4.pc /Users/mk/pkgsrc/wip/vis-editor/work/.buildlink/lib/pkgconfig/lua.pc
===> Configuring for vis-editor-0.8
=> Checking for portability problems in extracted files
checking for C compiler... clang
checking whether C compiler works... yes
checking whether compiler accepts -Werror=unknown-warning-option... yes
checking whether compiler accepts -Werror=unused-command-line-argument... yes
checking whether linker accepts -Werror=unknown-warning-option... no
checking whether linker accepts -Werror=unused-command-line-argument... no
checking whether compiler accepts -pipe... yes
checking whether compiler accepts -O2... yes
checking whether compiler accepts -ffunction-sections... yes
checking whether compiler accepts -fdata-sections... yes
checking whether linker accepts -Wl,--gc-sections... no
checking whether compiler accepts -fPIE... yes
checking whether compiler accepts -fstack-protector-all... yes
checking whether linker accepts -Wl,-z,now... no
checking whether linker accepts -Wl,-z,relro... no
checking whether linker accepts -pie... no
creating config.mk... done
checking for pkg-config... yes
checking for libcurses...
 checking for ncursesw... no
 checking for ncurses... yes
checking for libtermkey... yes
checking for libtre... no
checking for liblua >= 5.2 ...
 checking for lua... yes
checking for Lua statically linked liblpeg...
 checking for static lpeg... no
 checking for static lua5.3-lpeg... no
 checking for static lua5.2-lpeg... no
checking for memrchr... no
completing config.mk... done
===> Building for vis-editor-0.8
=> Fixing binary names.
WARNING: [subst.mk:binary-names] Nothing changed in "lua/plugins/filetype.lua".
WARNING: [subst.mk:binary-names] Nothing changed in "lua/plugins/number-inc-dec.lua".
WARNING: [subst.mk:binary-names] Nothing changed in "lua/plugins/textobject-lexer.lua".
=> Fixing absolute paths.
cp config.def.h config.h
clang -O2 -I/opt/pkg/include/lua-5.4 -I/opt/pkg/include/readline -I/opt/pkg/include/ncurses -D_XOPEN_SOURCE_EXTENDED=1 -I/opt/pkg/include -pipe -O2 -ffunction-sections -fdata-sections -fPIE -fstack-protector-all -I/opt/pkg/include -D_DARWIN_C_SOURCE -I/opt/pkg/include/ncurses -I/opt/pkg/include     -I/opt/pkg/include/lua-5.4 -DLUA_COMPAT_5_1 -DLUA_COMPAT_5_2 -DLUA_COMPAT_5_3 -DLUA_COMPAT_ALL  -std=c99 -D_POSIX_C_SOURCE=200809L -U_XOPEN_SOURCE -D_XOPEN_SOURCE=700 -DNDEBUG -D_FORTIFY_SOURCE=2 -D_DARWIN_C_SOURCE -DVERSION=\"0.8\"  -DHAVE_MEMRCHR=0 -DVIS_PATH=\"/opt/pkg/share/vis-editor\" -DCONFIG_HELP=1 -DCONFIG_CURSES=1 -DCONFIG_LUA=1 -DCONFIG_LPEG=0 -DCONFIG_TRE=0 -DCONFIG_SELINUX=0 -DCONFIG_ACL=0  array.c  buffer.c  libutf.c  main.c  map.c  sam.c  text.c  text-common.c  text-io.c  text-iterator.c  text-motions.c  text-objects.c  text-util.c  ui-terminal.c  view.c  vis.c  vis-lua.c  vis-marks.c  vis-modes.c  vis-motions.c  vis-operators.c  vis-prompt.c  vis-registers.c  vis-text-objects.c  text-regex.c -L/opt/pkg/lib  -L/opt/pkg/lib -ltermkey -L/opt/pkg/lib -Wl,-search_paths_first -lncurses     -L/opt/pkg/lib -llua5.4 -lm  -lc -o vis
clang -O2 -I/opt/pkg/include/lua-5.4 -I/opt/pkg/include/readline -I/opt/pkg/include/ncurses -D_XOPEN_SOURCE_EXTENDED=1 -I/opt/pkg/include -pipe -O2 -ffunction-sections -fdata-sections -fPIE -fstack-protector-all -std=c99 -D_POSIX_C_SOURCE=200809L -U_XOPEN_SOURCE -D_XOPEN_SOURCE=700 -DNDEBUG -D_FORTIFY_SOURCE=2 -D_DARWIN_C_SOURCE -DVERSION=\"0.8\"  vis-menu.c -L/opt/pkg/lib -lc  -o vis-menu
clang -O2 -I/opt/pkg/include/lua-5.4 -I/opt/pkg/include/readline -I/opt/pkg/include/ncurses -D_XOPEN_SOURCE_EXTENDED=1 -I/opt/pkg/include -pipe -O2 -ffunction-sections -fdata-sections -fPIE -fstack-protector-all -std=c99 -D_POSIX_C_SOURCE=200809L -U_XOPEN_SOURCE -D_XOPEN_SOURCE=700 -DNDEBUG -D_FORTIFY_SOURCE=2 -D_DARWIN_C_SOURCE -DVERSION=\"0.8\"  vis-digraph.c -L/opt/pkg/lib -lc  -o vis-digraph
*** Please use pkgtools/verifypc to sanity check dependencies.

Sorry for the noise - will fix the port and push to main...