Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multibyte languages rendering is strange(9.0.472) #1296

Closed
7 tasks
shikato opened this issue Sep 26, 2022 · 20 comments · Fixed by #1297
Closed
7 tasks

multibyte languages rendering is strange(9.0.472) #1296

shikato opened this issue Sep 26, 2022 · 20 comments · Fixed by #1297
Milestone

Comments

@shikato
Copy link

shikato commented Sep 26, 2022

Steps to reproduce

  1. Launch MacVim 9.0.472
  2. Input multibyte languages
    • e.g.
      • こんにちは
      • 안녕하세요
  3. The text rendering is so strange.

スクリーンショット 2022-09-26 23 25 16

Expected behaviour

I expect normal rendering.

スクリーンショット 2022-09-26 23 27 19

Version of Vim and architecture

9.0.472

Environment

macOS 12.3.1 (21E258)

How MacVim was installed

No response

Logs and stack traces

No response

Vim configuration where issue is reproducable

No response

Issue has been tested with given configuration

  • by running MacVim.app from GUI macOS interface
  • by running vim/gvim/etc installed by MacVim
  • by running other versions of vim (e.g. /usr/bin/vim)

Issue has been tested with no configuration

  • by running mvim --clean (or gvim, supplied by MacVim distribution)
  • by running vim --clean (in terminal, supplied by MacVim distribution)
  • by running vim --clean (in terminal, other suppliers, e.g. /usr/bin/vim)

Other conditions

  • The both Homebrew packages "vim" and "macvim" are installed
@shikato
Copy link
Author

shikato commented Sep 26, 2022

I checked that the bug do not occur in version 9.0.0065.
https://github.com/macvim-dev/macvim/releases/tag/snapshot-173

@eirnym
Copy link
Contributor

eirnym commented Sep 26, 2022

I have a test for various characters in my vimrc (demonstration you see on a screenshot below). My guess it is connected with drawing emoji and other wide characters from fallback font in general.

As far as I remember (as it was discussed before at least), the rectangle was calculated from the main font and to other fonts were cut as they were wider. newer implementation just doesn't cut the character size from fallback font and you see a mix. Long story short, you can control if main font is monospaced, but you can't control spacing from another fonts. Also you can't check and control all characters in a whole document you edit as it could be way bigger than you observe. (I advocate here to Vim's implementation, where what you see on a screen is what Vim renders, and it renders a very small portion of it. Because of clever implementation, Vim can easily edit very huge files without a penalty)

I personally prefer this implementation than a lot of font and document hacks.

In screenshot below, rightmost right arrow on a line above emoji takes the same width as w above as these symbols are taken from Fira Code Light I use. On the other hand, some non-English, Japanese, "Powerline" and Emoji characters are taken from fallback system fonts, so could have their own width (which is not exactly the same as English letters). You can see this discrepancy in "Powerline" section: lines are looks different despite they have exactly the same amount of characters in them. Similar situation you'll see in emoji: first emoji is in the same column as arrow I mentioned above, but way wider.

Thanks to a good implementation, movement works "visually" correctly: despite every japanese hieroglyph is a single character, when I move up from the second, my cursor will move to ι, not to ε.

Zrzut ekranu 2022-09-26 o 22 53 12

@eirnym
Copy link
Contributor

eirnym commented Sep 26, 2022

I described how I remember development of MacVim and my personal opinion. It could be different from yours as I don't type often in other languages than European. I even don't have special fonts for eastern languages

@ychin
Copy link
Member

ychin commented Sep 27, 2022

I can't really reproduce this. Some questions for you @shikato:

  1. Have you run this with mvim --clean as prompted by the issue template? It makes sure no special guioptions are set.
  2. Also can you show me what this command in Terminal shows (defaults read org.vim.MacVim | grep MM)?
  3. Related to (2), but do you have Core Text Renderer disabled?
  4. What system languages do you have set, and their relative order? In particular, I'm asking for the list of languages specified in System Preferences → Language & Region. The languages there actually affect the way the OS substitutes fonts as @eirnym was alluding to above.
  5. What happens if you put a cursor on the third and fourth column? What's in between them? Does the cursor become super wide?

I don't think any code in the relevant parts had changed in the release so there shouldn't have been much change, so I'm a little surprised to see any rendering differences.

@ychin
Copy link
Member

ychin commented Sep 27, 2022

In screenshot below, rightmost right arrow on a line above emoji takes the same width as w above as these symbols are taken from Fira Code Light I use. On the other hand, some non-English, Japanese, "Powerline" and Emoji characters are taken from fallback system fonts, so could have their own width (which is not exactly the same as English letters). You can see this discrepancy in "Powerline" section: lines are looks different despite they have exactly the same amount of characters in them. Similar situation you'll see in emoji: first emoji is in the same column as arrow I mentioned above, but way wider.

The width of a character is always either 1 or 2 in Vim. It's calculated by Vim (not MacVim), and not determined by the font at all. In fact, if the two (Vim versus font) don't agree, that's usually when you would get weird artifacts.

@eirnym
Copy link
Contributor

eirnym commented Sep 27, 2022

The width of a character is always either 1 or 2 in Vim. It's calculated by Vim (not MacVim), and not determined by the font at all. In fact, if the two (Vim versus font) don't agree, that's usually when you would get weird artifacts.

Thank you for describing this, I took it from "cut emoji" issue.

@shikato
Copy link
Author

shikato commented Sep 27, 2022

1

I tried mvim --clean, but same issue occurred.
スクリーンショット 2022-09-27 19 38 21

@shikato
Copy link
Author

shikato commented Sep 27, 2022

2

スクリーンショット 2022-09-27 19 45 23

@shikato
Copy link
Author

shikato commented Sep 27, 2022

3

Core Text Renderer off

The issue occurred.
スクリーンショット 2022-09-27 19 44 16

Core Text Renderer on

The issue did not occur. (I noticed it in this test.)
スクリーンショット 2022-09-27 19 44 42

@shikato
Copy link
Author

shikato commented Sep 27, 2022

4

language order

  1. Japanese
  2. English

and location is Japan.
スクリーンショット 2022-09-27 19 53 38

@shikato
Copy link
Author

shikato commented Sep 27, 2022

5

The explanation is very difficult, so I recorded a video.
Can you watch it? @ychin
https://user-images.githubusercontent.com/4592677/192508749-0b0a823d-74b1-4483-8948-2afbec7b9bbb.mov

@eirnym
Copy link
Contributor

eirnym commented Sep 27, 2022

@shikato yes, video explained issue quite better

How did you install MacVim? I have MacVim installed from GitHub releases and have no such issue via copy and paste and direct input (I installed standard japanese keyboard for the test via system preferences)

@shikato
Copy link
Author

shikato commented Sep 27, 2022

@eirnym
I installed MacVim from github release notes dmg file's link

I found a tweet, maybe it is same issue, too.
https://twitter.com/sat_toke/status/1572623078760521730

@ychin
Copy link
Member

ychin commented Sep 27, 2022

@shikato , out of curiosity, why are you setting Core Text renderer to off? As the description there says, this is deprecated, meaning that using MacVim without Core Text renderer is no longer supported (and we no longer fix bugs / glitches associated with it). I still exposed the option for now in case we need to debug issues, but it would be removed in the near future with no option to turn it off. Can you just use it with Core Text renderer on? What feature do you find missing?

@ychin
Copy link
Member

ychin commented Sep 27, 2022

I'm still curious what broke though, so I'll take a look, but you should probably assume that Core Text Renderer off will break at any time.

@ychin
Copy link
Member

ychin commented Sep 27, 2022

I think this was caused by #1287 just for reference.

ychin added a commit to ychin/macvim that referenced this issue Sep 28, 2022
This was previously broken by macvim-dev#1287 as I did not update the new
constants' values.

Fix macvim-dev#1296
@shikato
Copy link
Author

shikato commented Sep 28, 2022

@ychin I have a hazy memory, I think the reason was to avoid similar issue.
#1168 (comment)

I think the issue doesn't occur in current version, so I can enable Core Text renderer.

Thank you for confirming.

@ychin
Copy link
Member

ychin commented Sep 28, 2022

Right. Yes please enable Core Text renderer. I will fix this bug anyway since it's easy to fix, but I should probably just remove the option to unset Core Text renderer in a future release to avoid stale settings like this leading to obscure bugs. Feel free to file additional issues if you find other issues.

ychin added a commit to ychin/macvim that referenced this issue Sep 28, 2022
This was previously broken by macvim-dev#1287 as I did not update the new
constants' values.

Also, make strikethrough work in it.

The non-CoreText renderer is essentially deprecated and will be removed
in near future, but for now just fix up the straggling bugs since this
is easy to do so.

Fix macvim-dev#1296
@ychin ychin added this to the Release 175 milestone Sep 28, 2022
@sattoke
Copy link

sattoke commented Oct 3, 2022

@shikato

I found a tweet, maybe it is same issue, too.
https://twitter.com/sat_toke/status/1572623078760521730

Thank you for referring to my tweet.

@ychin
I enabled "Use Core Text renderer" and it solved the problem in my environment as well! Thank you!

In my environment, with that option enabled, MacVim (I don't remember the version) did not display properly in the former Big Sur environment, such as the display blinking, so I disabled it.
https://twitter.com/sat_toke/status/1368800041708650497

@ychin
Copy link
Member

ychin commented Oct 3, 2022

I see. Thanks for the context. I'm probably going to remove the option to unset the core Text renderer in the future. If we run into more displays issues we will just have to fix them instead of allowing a poor workaround (I still think it was likely due to the MacVim version being old and didn't have the latest fixes to the renderer).

ychin added a commit that referenced this issue Feb 7, 2023
Updated to Vim 9.0.1276

Features
====================

Dictionary lookup
--------------------

You can now use Force Touch or Cmd-Ctrl-D to look up definitions of word
under the cursor (or selected text in visual mode). This will also
preview URLs, and support data types such as phone numbers and
addresses. #1312 #1313

This feature can also be invoked programmatically from VimScript (see
`:h macvim-lookup`). #1315

Tool bar / Touch Bar / menu icons
--------------------

You can now use SF Symbols for Tool bar and Touch Bar icons, including
using different symbol styles such as "palette" or "multicolor". Menu
items can now also use the `icon=` syntax to specify icons as well. See
`:help macvim-toolbar-icon` for details. #1329

The default tool bar also has updated icons to look similar to SF
Symbols used by newer macOS versions. #1214 by @sfsam

Window management actions
--------------------

There are new `macaction`'s for managing the MacVim window. The new
`zoomLeft`/`zoomRight` actions allow you to pin the window to the
left/right of the screen, and there are also new actions for interacting
with Stage Manager (requires macOS 13+). See `:h macvim-actions` for
details. #1330

Pre-release updates / Sparkle 2
--------------------

MacVim now supports pre-release software builds. It's sometimes hard for
us to release frequent updates due to the desire to pick a stable
upstream Vim version, needing to test the release on multiple OS
versions, making sure there aren't half-complete or buggy features, and
other reasons.

This new feature now allows us to push pre-release beta builds out in a
more frequent fashion, which could be useful if there are particular
features or fixes that you would like to try out before the next
official release. Pre-release builds will be released depending on bug
fixes and features instead of a fixed cadence. Do note that these
pre-release builds may not be as well-validated and may have half-baked
features.

If you are using the built-in auto-updater to update MacVim, you can
turn this on by going to Advanced settings pane, and enable "Enable
pre-release software updates".

This feature is only available for macOS 10.13 or above.

The auto-updater has also been updated from Sparkle 1.27.1 to 2.3.0 for
10.13+ builds. Legacy (10.9-10.12) builds are still using Sparkle 1.

See #1332.

New Vim features
--------------------

New `smoothscroll` option allows you to scroll through a long wrapped
line (using Ctrl-E or mouse wheel) without immediately jumping to the
next line. (v9.0.0640)

`splitscroll` option has been renamed `splitkeep`, with more flexibility
than before. (v9.0.0647)

Sound playback on macOS is now supported. You can use `has('sound')` to
check. See `help sound` for details. (v9.0.0694)

Terminals now support `:confirm` for `:q`, etc, which also means
MacVim's Cmd-W will work properly for terminal windows. (v9.0.0710)

Virtual text had numerous bugs fixed.

General
====================

Legacy build for 10.9 - 10.12
--------------------

Per a previous announcement (#1271), the default MacVim binary will now
require macOS 10.13 or above. Users of macOS 10.9 - 10.12 can use a
separate "legacy" build which will still be supported. The legacy binary
will still have the latest versions of Vim and be supported, but may not
have all the latest features (e.g. pre-release builds).

If you are using the auto-updater (Sparkle) to update MacVim, it should
"just work" and find the best version for you. If you are downloading
MacVim from the website, there is also a link to download the legacy
version marked for 10.9+ as well. If you download the normal binary
marked for 10.13+ from the website, it won't work on these older macOS
versions.

See #1331.

Fixes
====================

CoreText Renderer clipping and rendering bugs
--------------------

Unicode characters with multiple composing characters (e.g. "x⃗") will
now render correctly. #1172

Texts (e.g. Tibetan, Zalgo texts) that are taller than the line height
will no longer be clipped inappropriately. You can use a new setting
`MMRendererClipToRow` to re-enable clipping if the tall texts are
distracting. #995 / #1356

Tab crash
--------------------

Fixed a crash when opening new tabs that seems to only occur in macOS 13
Ventura. #1333

Other bugs
--------------------

- Fixed non-native full screen not working well with the notch on newer
  MacBook's when set to not show menu bar. You can also use
  `MMNonNativeFullScreenSafeAreaBehavior` to force MacVim to use the
  notch area as well if you don't mind some content being obscured. Note
  that the previous release also claimed it fixed this, but because the
  binary was built against an old macOS SDK (Big Sur), the fix did not
  work in the binary release. #1261
- Allow "Open untitled window: never" and "After last window closes:
  Quit MacVim" to be set together again. Added safeguards to make sure
  doing so won't immediately close the app. #1338
- Edit.Cut / Copy menu items will now be properly disabled when there
  isn't selected text. #1308
- Fixed potential `:emenu` crash when the menu is associated with an
  action in a non-valid mode. #1305
- Fixed bug where just bringing up the right-click (or the
  MacVim→Services) menu would somehow copy the selected texts to the
  system clipboard. #1300
- Fixed a Japanese input method bug where using left/right arrow to move
  to a different section of the input text would previously result in
  the candidate list not showing up at the correct position. #1312
- Fix non-CoreText renderer not handling text styles like strikethrough
  correctly (note: this renderer has been deprecated for a while and you
  should not use it). #1296
- This release uses an older sh/bash syntax file because the latest one
  in Vim has a bug. #1358

Misc
====================

New settings:

- "No drop shadows" (Appearance). #1301
- "Treat Ctrl-click as right-click" (Input) (#1326). This was previously
  configurable via command-line, but now also possible in the settings
  pane under the new "Input" category.

"About MacVim" now reports the version number in a clearer way with
clearly specified release number vs Vim version.

Known Issues
====================

Printing
--------------------

Printing using File→Print or `:hardcopy` is currently not working under
macOS 13 Ventura due to its removal of PostScript support in the Preview
app. This will be fixed in a later release. See the issue for
workarounds. #1347

Scripting
====================

- Scripting languages versions:
    - Perl is now built against 5.30, up from 5.18.
    - Ruby is now built against 3.2, up from 3.1.

Compatibility
====================

Requires macOS 10.9 or above. (10.9 - 10.12 requires downloading a
separate legacy build)

Script interfaces have compatibility with these versions:

- Lua 5.4
- Perl 5.30
- Python2 2.7
- Python3 3.10
- Ruby 3.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants