-
Notifications
You must be signed in to change notification settings - Fork 274
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: unicode truncation bug #1089
Conversation
Due to the truncation the non truncation path used to_owned which cloned the line. Calling an inner function on both cases resolves this as the temporary truncated line lives long enough for that function to end.
This has 3 bugs still I think in the rendering, but these are not panics. Fixes ratatui-org#1032
This is introduced as a helper for a bug fix and I'm not 100% sure it is correct (and it should have tests to verify it). So for now, I'm making it `pub(crate)` to avoid exposing it to the public API.
I'd suggest that for this PR it's probably worth:
Note this stems from #1082 credit goes to @EdJoPaTo for making it easier to see a way to make a fix for this. Side note: there's also a truncate crate, which does various unicode split_at things, but it uses |
I also suggested unicode-segmentation for unicode-truncation as it currently rips apart stuff that belongs together like Emoji combinations (Flags for example): Aetf/unicode-truncate#11 |
There's an EXTREMELY lengthy debate on to use / not use segmentation in #75 (ratatui's longest open issue). I don't know which side of that debate has more merit, and it's likely that using unicode-segmentation within the truncation crate would break things in other ways for many terminals / use cases. |
Also, I forgot that we do use segmentation when doing grapheme splitting, so maybe that is the right abstraction. I'm unsure how exactly to test an edge case on this one without delving a vast amount more into unicode than I want to right now. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1089 +/- ##
=====================================
Coverage 94.1% 94.1%
=====================================
Files 61 61
Lines 14619 14664 +45
=====================================
+ Hits 13764 13809 +45
Misses 855 855 ☔ View full report in Codecov by Sentry. |
This comment was marked as outdated.
This comment was marked as outdated.
The code is very readable and merging this even with bugs is good to me! I like that there are tests that capture the behavior. I don't quite understand why this works (haven't had a lot of time to go through it in detail). Maybe the PR description can have a high level description of why this works? Unrelated to this PR but it would nice to collect links to resources on the developer guide sector of website about Unicode that for uninitiated potential contributors to read. |
49e7e0c
to
dda2106
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Co-authored-by: EdJoPaTo <github@edjopato.de>
Co-authored-by: EdJoPaTo <github@edjopato.de>
Added heavily inspired by unicode-truncate code to find the index in order to get the starting index. This allows for taking a reference rather than cloning into an ever-more-allocating String. I think this would be a great addition for unicode-truncate especially with all the related test cases specific for this. Some test cases expect (Mentioning @Aetf here as they might be interested in this discussion) Did not remove the split_at method which obviously annoys the CI. Benchmark after the change. (Results on a Raspberry Pi 3 are similar while taking ~10 times the time)
|
Oh… it actually doesn't in our use case as we can just increase the |
I'll wait on @EdJoPaTo's approval to merge this |
This comment was marked as outdated.
This comment was marked as outdated.
142295e
to
9de47c5
Compare
The logic is somewhat horribly complex as a lot of stuff isn't intuitive… The widths are usize and not u16 as the truncation of the end is implicitly done. That resulted in misunderstandings in what should happen. I don't think we can fully explain the code by comments. We need many tests to ensure it actually does what it is expected to do. Lines/Spans longer than u16 for example is a whole new mess which I haven't thought about before. Also, I refactored the code to iterator logic. I'm fine with reverting it. Benchmarks suggest it's similar in performance. But it might be even harder to follow what is going on? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It somewhat worries me that new test cases pop up and break stuff even when we thought we are done and only need documentation stuff. But it's way better than the current state of random panics.
The approach of debug_asserts is somewhat I like here as it ensures that assumptions are actually true and documents these assertions in the process. And they only panic on debug, so production code will only be displaying stuff wrong, not panicking. Also, we have many test cases and users of ratatui will provide more when they run their stuff as dev builds.
So… documentation won't be ever perfect, and I kinda expect new cases that break the current code as Unicode is Unicode. So whoever wants to improve this should have many test cases available. Which is what we achieved here for sure.
So I'm in for merging this in order to have a better state than before. We won't reach a perfect state anyway.
its 8% to 30% faster in the benchmark
- Remove perf fast path for left aligned lines as the absolute perf diff is negligible (30ns on an M2 Mac) - move the visible spans calculation to a method that returns an iterator over the spans that are visible - replace truncation clippy lints with converions that clamp to u16::MAX - replace arithmetic_side_effects clippy lint with saturating_sub - remove debug_asserts - rewrite comments in the imperative mood
Some final tweaks: It's important to look the absolute magnitude of a perf gain, and not just the relative amount. On my M2 mac, removing the fast path costs ~30ns. For an app running at an absolutely absurd 1000FPS, the app would now run at 999.97 FPS. This isn't worth keeping the code to optimize.
|
I'd like to get this released (perhaps Monday if @orhun is up to it). Direct link to the current file as of the last commit: Line 555 in 43118c5
I'm not too worried - most text that's rendered in the context of a tui app isn't > u16::MAX, and when it is the resultant bug is easy to fix by pre-trimming the text instead of doing this in the rendering. (There's an actual bug in turbo that is similar to this problem, but with
The complexity of the implementation is now pretty much aligned with the inherent complexity of the problem, and it's documented well enough.
I replaced these with infallible code. I don't like debug_asserts at all, especially in a library. When you're wrong and they fail, you're slapping your user's users with a pain point which is hard to report, and which takes a release of both projects in order to fix. |
I am not worried about the actual bug that came up. I am more worried that 3 people looked over it for days and missed yet another bug.
What I liked about it were the assumptions actually tested. We wrote code with assumptions which are now hidden behind infallible code. But with all the tests, fine by me.
I am not at all worried about way overpowered hardware. I know that Line is one of the core constructs used by a lot of things. Relatively small improvements will impact a lot of code depending on it. So it's way more significant to improve the performance here. And also the Even when it's not about lower power devices or hardware that is years old (My main device is ~9 years old now). It saves CPU cycles and therefore battery life. |
To pick even more on the performance fast path of Let's imagine a table rendered on a
And this is a Raspberry Pi 2 benchmark, I use Even an improvement of 12% is really good here already. The benchmark goes to a regression of 36%. So this is definitely significant. For the default of left alignment which is the most often used one. Benchmark on Raspberry Pi 2:
Benchmark of Raspberry Pi 4: regression of 14.3 to 38.7%. Worst case takes I will not benchmark this on Pi 1 as running the benchmarks on Pi 4 take ~10 min, on Pi 2 ~40 min. Pi 1 is single core in comparison to Pi 2 with 4 cores. |
Let's look at the absolute magnitude of the change though: (10 * .125) = 1.25us per call 1.25*450 = 0.563ms, so each frame now takes (16.667+0.563) = 17.23ms = 58 FPS. That's measurable sure, but rarely directly noticeable. Assuming a factor of 4 on the Pi 1, 18.92ms = 53 FPS. One thing I did note when removing the fast path was that the slow paths got slightly faster (1-4%), so the tradeoff is one that doesn't necessarily have a definitively good answer. For now, I'd like to keep it simplified, and keep this PR about fixing the bug and not trying to eek out all the performance of this. You've convinced me that there are some parts of this that matter to you, so let's continue the discussion about perf in an issue / forum topic if it's more a discussion.
This raises an interesting point. I wonder (not for this PR), what parts of our rendering pipeline might be able to be done more in parallel. |
Let's move the perf discussion to another Issue for discussion later
- Rewrote the line / span rendering code to take into account how multi-byte / wide emoji characters are truncated when rendering into areas that cannot accommodate them in the available space - Added comprehensive coverage over the edge cases - Adds a benchmark to ensure perf Fixes: ratatui-org#1032 Co-authored-by: EdJoPaTo <rfc-conform-git-commit-email@funny-long-domain-label-everyone-hates-as-it-is-too-long.edjopato.de> Co-authored-by: EdJoPaTo <github@edjopato.de>
ratatui fixed [unicode truncation bug](ratatui-org/ratatui#1032) in ratatui-org/ratatui#1089
Fixes: #1032