Flag emojis display as squares

[llimllib]

Describe the bug

Given this text file as t.txt:

* 🇺🇸 
* 🇰🇷 

on mac ventura, neovim displays the flag emojis incorrectly on iterm2 and terminal.app, but correctly on kitty.

iterm and terminal.app themselves have no trouble displaying the emoji.

nvim -u none t.txt (v0.8.2 from homebrew) on iterm (3.4.19, the latest as of right now), looks like this:

while cat t.txt shows it properly:

nvim -u none t.txt on terminal.app looks similarly:

but on kitty (0.26.5), it works properly:

I tried a few TERM values, as I noticed kitty had TERM=xterm-kitty, but nvim on iterm failed with TERM set to all of: ansi-256color, xterm-256color, xterm-kitty and xterm-new

I reproduced this behavior on a second macbook with the same versions of these softwares.

All of these programs also display the flags correctly in iterm: bat, nano, emacs, helix

TextEdit, code and VimR do as well

Steps to reproduce

1. edit a file with neovim 0.8.2 that contains flag emojis on mac ventura, with iterm 3.4.19

Expected behavior

I expect it to show flag emojis

Neovim version (nvim -v)

NVIM v0.8.2

Vim (not Nvim) behaves the same?

yes, version 9.0

Operating system/version

macOS 13.1 ventura

Terminal name/version

iterm 3.4.19, see above for further details

$TERM environment variable

xterm-256color

Installation

homebrew

[zeertzjq]

Related: #7151

[llimllib]

If you have ideas for further information I could get you, by running a beta or debugging, let me know, I'd be happy to dig in further

[WhyNotHugo]

Same issue on foot on swaywm/wayland.

FWIW, there's some background in #16447, though since have [very slightly] changed since.

[llimllib]

I'm going to close this in favor of #7151, because I believe that this is an instance of that and there's no need to have a duplicate issue here.

It makes me sad that this will take so long to fix! Not that I think it's anybody's fault it's a very complex problem, I just wish we'd converge on libraries to make it easier.

[zeertzjq]

No, this is a bit different as a flag emoji consists of two characters of the same type without a separator.

[llimllib]

ok, you know better than me! Thanks for all the work on neovim, I appreciate it