Open-sourced your benchmark?

[Naereen]

Hi,

• First, I just tested your project on XUbuntu 16.10, and everything works fine. Good job, it's easy to install and launch.
• I think your warning about the config file in the README.md file might not be necessary (as you say, if the config file is not present, it is created on the first run of alacritty).

I tested alacritty's performance against xfce4-terminal, on simple programs, and on cmatrix. My observation is that runs slightly slower than xfce4-terminal.

But it is unclear how you measured the performance and what kind of benchmarks did you use.
Could it be possible to explain more, and maybe open-source whatever software or script was used for the benchmark?

I would be curious to test again, with a more robust measure.
Thanks.

[elopez]

I'd like to know how performance was measured too. I ran a non-scientific benchmark: executing "time find /usr" a bunch of times on an Arch Linux machine with an SSD. Alacritty took about 40s to print all of it, while GNOME Terminal always finished in under 3s. Even xterm did better, doing it under 6s.

[jwilm]

Benchmarks so far have just involved running find /usr on my Linux system with Alacritty, st, and urxvt, and on macOS against Terminal.app and iTerm2.

[alexhultman]

maybe it is not a good idea to claim to be fastest (it is a superlative) when you havent tested more than 3 projects.

[MoSal]

Benchmarks so far have just involved running find /usr on my Linux system with Alacritty, st, and urxvt, and on macOS against Terminal.app and iTerm2.

That's hardly an extensive test scenario. And an odd choice of GNU/Linux terminals.

find /usr doesn't cover heavy usage of many codes. Testing with saldl , for example, immediately exposes color issues, flashing every few seconds, among other issues. Those flashes are not seen when using vte-based terminals, konsole, mlterm, or OSX's Terminal. But I do vaguely remember seeing them in ConEmu, and maybe other slow emulators.

While the potential is there, I don't think Alacritty can be called a fast, let alone the fastest, terminal emulator at the moment.

[Naereen]

I agree with @MoSal.

[roobie]

Commands one can use to generate output:

• dd if=/dev/urandom | base64
• cd /; tree

I tested these with Terminator, Sakura and Alacritty, and Alacritty performed very well

EDIT: Dunno if executing inside a tmux session makes a difference, but I tested that as well, which yielded good results.

[jwilm]

I'd like to apologize to those of you who feel mislead about performance claims. You're right that 4 comparisons is not particularly thorough given the volume of terminal emulators on the market. FWIW, I've been doing these comparisons all through development and found Alacritty to be the fastest.

Moving forward, I plan to continue marketing Alacritty as the fastest terminal emulator available and back that up with indisputable benchmarks. It seems like there's a bit of work to do on that front.

while already at it, isnt every terminal emulator "GPU accelerated"? i find it very hard to believe gnome terminal, as an example wouldnt use the graphics card at all. seems kind of strange to make a big deal out of utilizing the graphics card to draw graphics in 2017 - but maybe im wrong?

Assuming for a moment that X drawing APIs use the GPU wherever possible, drawing APIs like that are nearly always done in immediate mode. This mode is incredibly wasteful in the context of a GPU due to frequent uploads, lack of batching, and frequent state changes.

Contrast this with what Alacritty does--retained mode rendering. Display lists for what's on the screen are precomputed from the terminal state. Those lists are uploaded to the GPU, and then everything is drawn at once using only two draw commands.

[Naereen]

Moving forward, I plan to continue marketing Alacritty as the fastest terminal emulator available

It's good to stay positive and motivated about your project, although I really think a serious and complete benchmark is needed (e.g., with plots and tabs of time values, for several heavy commands, as well as CPU/GPU/RAM usage for more than 10 emulators).

One reason for asking for more than find / or cd / ; tree examples is that these examples do not use ANSI color codes, do not use non-ASCII symbols or glyphs.

See for instance the benchmarks displayed on Julia's homepage.

[ghost]

@Naereen actually tree will use LS_COLORS if it's set.

Also don't forget about caches when benchmarking. Consider the following script where alacritty is sure to beat urxvt.

#!/bin/bash
time urxvt -e find /
time alacritty -e find /

A more fair benchmark would be:

#!/bin/bash
time urxvt -e find /
echo 3 > /proc/sys/vm/drop_caches
time alacritty -e find /

[lukaslueg]

We simply need some real #[benchmark]s in alacritty. They can include synthetic tests ("parse 100mb of random stuff", "draw the parsed output of 100mb of random stuff offscreen") and cooked up tests ("here is what the usual output on 12 panes of colored Vim does to you!"). It's the only way to tell if performance increases or decreases over time.

[theduke]

For reference, I'm posting my results here, including a good method of generating random data.

# Generate 100MB of random tex.
cat /dev/urandom | base64 | dd of=/tmp/randomdata bs=1024 count=100KB

# Test. Repeat for each terminal...
# Results are written to bench_XXX.txt
# This also shows MB/s very nicely thanks to `dd`.
{ time dd if=/tmp/randomdata bs=10240; } 2> bench_XXX.txt

My results, on Linux Arch with proprietary Nvidia drivers:

• Terminator: 4.4 MB/s
• urxvt: 19.3 MB/s
• alacritty: 24.5 MB/s

[MoSal]

Here is a more useful test:

while true; do
  sleep 0.01
  echo -e '\r'
  echo -e '\033[0K\033[1mBold\033[0m \033[7mInvert\033[0m \033[4mUnderline\033[0m'
  echo -e '\033[0K\033[1m\033[7m\033[4mBold & Invert & Underline\033[0m'
  echo
  echo -e '\033[0K\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[30m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[30m\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A'
done

Reduce the sleep interval until the cursor movement becomes obvious. Or remove the sleep, and the last echo, and repeat for X times.

I'd suspect the raw performance will greatly vary depending on the GPU/CPU combination.

For me, the major issue is the flashing/blinking, which I'm seeing every few seconds in alacritty, even with long sleep intervals.

(Off-topic: With zero and positive y offsets. Text seems to be bottom-adjusted instead of centered in each line. Is the default negative to work around this known issue?)

[sijad]

alacritty feel much faster to me:

browsing ~/.config/alacritty/alacritty.yml with vim 8 (default configuration), just keeping up arrow (k) button done.

[quininer]

I have an intuitive way to test terminal performance.
use mpv --vo tct to play a video and see if it is dropped frame.

[kutsan]

I agree, claiming that it's the fastest terminal emulator out there, is questionable. Using system binaries like find does not produce consistent result across other systems. There are so much factors that affect its benchmark. Even the version of find used to test. You should frankly and clearly demonstrate why it's the fastest and how. It would be nice to create a platform agnostic benchmark script to test it. So, people can run and compare their results in different terminal emulators and your results.

[tokudan]

seq from GNU coreutils generates a simple stream of numbers and is available on most systems:
$ time seq 1 999999

My result with xfce4-terminal:
real 0m13,890s
user 0m0,020s
sys 0m0,542s

My result with alacritty:
real 0m0,524s
user 0m0,016s
sys 0m0,465s

[Naereen]

@tokudan : one command, on one machine, comparing only two terminal emulators, and ran only once is NOT at all a benchmark...

[kmicu]

But it is still a fact and still a data point which shows relative speed on some workstation.

Here is more data points (executed multiple times, but results were not consistent only in case of rxvt):

Shell: GNU bash 4.4.12
DDX: modesetting
CPU: i7-4702MQ
GPU: Intel HD Graphics 4600
Compositor: compton-0.1_beta2.5
I do not bother with listing next x things that could affect rendering speed.
Relative speed is important here.
Test command: time for i in {1..3}; do seq 1 999999; done

XTerm 330
visually: flashy and uneven, stroboscopic

real 3m2.195s
user 0m0.092s
sys 0m2.004s

Alacritty
visually: consistent, no problems
real 0m1.377s
user 0m0.075s
sys 0m1.299s

st 0.7
visually: consistent, no problems

real 0m4.116s
user 0m0.067s
sys 0m1.747s

uxrvt v9.22
visually: consistent, no problems

different results on each invocation
from:
real 0m1.081s
user 0m0.052s
sys 0m0.864s

to:
real 0m1.626s
user 0m0.079s
sys 0m0.942s

xfce4-terminal 0.6.3
visually: jumpy prompt

real 0m44.232s
user 0m0.128s
sys 0m3.342s

Creating a proper benchmark would be really difficult if you take into account that hardware AND drivers AND Xorg config can affect results. So at the same time on my workstation alacritty can be the fastest and on your workstation can be the slowest. Good luck with buying hundreds of computers to create a proper, honest, reproducible benchmark to judge if a terminal emulator is really the fastest in the whole universe (づ￣ ³￣)づ

[alexhultman]

Regarding my first comment about superlatives (in the top), I just looked at the front page and saw this as the very first text:

"Alacritty is the fastest terminal emulator in existence."

I don't know about you, but this makes me throw up in my mouth. First of all, this is socially embarrassing behavior. You cannot possibly know this to be true, and it is awfully boastful. Not only is it implying "in the world" but also "in the entire universe". This, simply is bullshittery x2000.

Secondly, people have reported all kinds of numbers showing that in some cases this simply isn't even true. Since you obviously had a successful initial marketing, maybe it is time to tone down this aggressive marketing a bit. It is tasteless and tacky.

[kmicu]

People here criticizing facts and data have no idea how difficult task is to create a benchmark for rendering and how useful it would be in the end. Your imaginary benchmark is not only affected by what you do in a benchmark but also by its environment. So DDX drivers, hardware, a compositor config, and so on. So even if you get some numbers on your precious benchmark they are meaningless for others if you do not have bit by bit perfectly reproducible hardware and software.

@alexhultman “Secondly, people have reported all kinds of numbers showing that in some cases this simply isn't even true” and at the same time you are ignoring the fact that majority of numbers reported here show that Alacritty has the fastest rendering. You did not provide any additional data point disproving that.

If you folks are so much offended by Alacritty’s claims then show us data and not only emotional comments. On my setup Alacritty is marginally faster than (inconsistent) uxrvt. So I have no reasons to complain. If you do not like proposed seq 1 999999 or mpv --vo tct tests then show us a better one. Do something useful. Any data point is worth more than your complaints.

[MoSal]

@kmicu
Can you benchmark alacritty and konsole with this test:

for i in {1..400000}; do
  echo -e '\r'
  echo -e '\033[0K\033[1mBold\033[0m \033[7mInvert\033[0m \033[4mUnderline\033[0m'
  echo -e '\033[0K\033[1m\033[7m\033[4mBold & Invert & Underline\033[0m'
  echo
  echo -e '\033[0K\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[30m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[30m\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
done

[fschutt]

for @MoSal s benchmark:

• konsole: real 0m38,987s, user 0m31,216s, sys 0m7,744s
• alacritty: real 0m40,352s, user 0m32,380s, sys 0m7,956s
• gnome-terminal: real 1m7,990s, user 0m33,292s, sys 0m9,012s
• xterm: more than 5 minutes

All terminals had the same size, laptop with SSD, Intel HD drivers on X11. I would not claim that alacritty is "the fastest in all cases", but it's pretty fast. However, we are currently arguing about a 1 second speed difference of terminal emulators in a synthetic benchmark with 400K iterations. There are more important things in life.

[MoSal]

@fschutt

However, we are currently arguing about a 1 second speed difference of terminal emulators in a synthetic benchmark with 400K iterations. There are more important things in life.

Fully agree*. I just provided this counter test-case because someone asked for one.

The issue I'm still seeing with alacritty is slow cursor movement, which would manifest as flashing
in slower hardware (test case in my earlier comment). The issue is less pronounced in my new hardware. But it's still there.

* FWIW, I'm not a KDE user. And I don't use konsole because I think it's the fastest. I use it because it's the only one with usable BiDi support (less smart, but works better than mlterm).

[kmicu]

@MoSal Here we go:

(the same workstation and software/config versions as in my previous post)

visually: consistent, no jumping
real 0m27.552s
user 0m21.436s
sys 0m6.115s
real 0m28.232s
user 0m21.887s
sys 0m6.344s
real 0m27.728s
user 0m21.480s
sys 0m6.244s

Konsole 17.08.1
visually: jumpy and uneven, it goes, stops for a moment and goes again and so on
real 0m28.957s
user 0m20.540s
sys 0m5.671s
real 0m29.755s
user 0m20.441s
sys 0m5.811s
real 0m29.838s
user 0m21.428s
sys 0m5.751s

st: a little bit jumpy, but ok
real 0m26.933s
user 0m20.946s
sys 0m5.927s
real 0m26.575s
user 0m20.726s
sys 0m5.826s
real 0m26.868s
user 0m21.003s
sys 0m5.819s

rxvt
visually: a little bit jumpy, but ok
real 0m30.615s
user 0m21.345s
sys 0m6.231s
real 0m31.122s
user 0m21.478s
sys 0m6.032s
real 0m31.437s
user 0m21.875s
sys 0m6.211s

I do not like this particular test because 100% CPU is taken also by a shell process (bash, zsh, etc). So we are not testing only rendering, but also some funky interaction with a shell.

PS
Please, keep in mind that above numbers apply only to my workstation: a custom Compton config, DDX modesetting, a custom Xorg 1.19.5 config, EGL version 1.4 (DRI2) and so on.
Fun fact: I use UXTerm as my daily driver, because of http://invisible-island.net/xterm/xterm.faq.html#compare_versions

[alexanderadam]

Wow, @kmicu your PS is gold. It would be nice to have this compatibility for alacritty as well.

[AlexisTM]

Tmux performance issue? What am I doing wrong?

$ time seq 1 999999

[...]

real    0m0.444s
user    0m0.025s
sys     0m0.419s

$ tmux

$ echo $TERM
alacritty

$ time seq 1 999999

[...]

real    0m5.444s
user    0m0.042s
sys     0m0.562s

[Swoorup]

This is bit worrying. I used alacritty because of claims of being the fastest and in rust. But it doesn't turn out to be the reality as the tests shown here.

[chrisduerr]

@Swoorup Make sure you use the scrollback PR when testing. In my tests alacritty is the fastest terminal emulator in all current benchmarks (except for unicode stuff but that's not really related to "raw speed"). Even on master the random write is significantly faster though, just the scrolling had some performance tweaks.

[vks]

Terminal latency should probably be considered as well, see for example these benchmarks: https://danluu.com/term-latency/

[fschutt]

In https://gitlab.com/anarcat/terms-benchmarks, alacritty is fairly good and if you'd trust that benchmark, then yes, alacritty can claim that it is the fastest terminal emulator. Also see the article on LWN: https://lwn.net/Articles/749992/ - I think it depends on which benchmark you believe and in the end, everyone has to decide for itself.

In some benchmarks, alacritty is truly the fastest emulator, in others it isn't. I mean, since alacritty is built from source, there isn't even a guarantee that we are all testing the same binary. I do think that "alacritty is the fastest" is a lie, since it's not the fastest in all cases and in all benchmarks. It's also not the fastest by a huge margin, so differences of a few milliseconds can be attributed to system hiccups, other processes running in the background, etc. Some emulators cheat by not updating the whole screen at once, etc. Benchmarking is hard. However it's "one of the fastest", so I'll let it slide, if it motivates him to continue working on it and making alacritty faster.

[chrisduerr]

You should also make sure that whenever benchmarking things like scrolling (yes for example) to make use of the scrollback PR, there were some significant speed enhancements made in that PR which lead to alacritty taking the lead over both urxvt and st on my machine.

Regarding random updates (so raw terminal throughput) alacritty seems to beat the competition by a fair margin.

[therealraven]

Alacritty makes the terminal experience significantly better than iTerm2 on Mac OS. Many people know this. Please don't let negative comments in threads like this one discourage you. The terminal you have built is indispensable to Mac OS users.

Benchmarks aside for a moment, on Mac OS simply scrolling in Vim in Tmux in Alacritty is much faster in Alacritty compared to Terminal.app or iTerm2. It has a real impact on common daily tasks.

[ngocphamm]

I don't know why but scrolling (holding down j/k) in Neovim, in Tmux using Alacritty and iTerm2 (3.1 or 3.2 beta now) doesn't feel much different to me, even with rendering using Metal OFF in iTerm2. Terminal.app does feel a little more sluggish, though.

Background: I have KeyRepeat set at 15ms.

[chrisduerr]

@therealraven I appreciate the support, I really do :), but at the same time, complaining about alacritty being slow is also support (when done in a respectable manner). Alacritty aims to be blazingly fast, so whenever it's not, that's a bug and reporting such a bug is a good thing. It's unfortunately impossible to test alacritty on every system so it can be kinda tough to troubleshoot and fix some issues without people complaining.

It's always great to hear positive comments, but I don't think (or at least I hope so) anyone is discouraged by valid criticism about shortcomings and bugs about alacritty, so no worries there. :)

Thanks for the feedback!
(Note: I'm just speaking for myself here, but I doubt jwilm or anyone of the other lovely contributors strongly disagrees with any points I've made)

[sandersantema]

In my experience alacritty is much faster than iTerm and certainly much faster than Terminal.app. Even if Alacritty wouldn't be faster than iTerm i'd still use it since:

• It's simple, which is a plus for me.
• It seems stable (even though I'm using a experimental branch)
• It's the only app which doesn't have those awful borders around the text which I despise so very much, together with the speed of Alacritty this was the biggest reason I've chosen to use Allacritty.

Alacritty is a fantastic application and I believe it only can become better 😄

[MoSal]

I wish macOS users would start including kitty when they are reporting their experiences.

I'm not a macOS user myself, so I have no idea how performant iTerm2 is. But I tested kitty in a GNU/Linux system, and it is a decent emulator.

[lompabo]

Hi all, At least on my Mac (high Sierra, MacBook Pro 2015), kitty (installed via homebrew) is way slower than alacrity. I made some tests by using “cat” a ~20MB text file and “time” to measure the performance. I didn’t save the numbers, but the speed ranking was: Alacritty > Terminal.app >> kitty > iTerm2 I was actually surprised by how much slower kitty turned out to be w.r.t. Terminal and alacritty (roughly 10 times slower). There’s always the possibility that it due to some local configuration quirk, but for me that test sealed the deal. Just my two cents, Michele Il giorno mer 23 mag 2018 alle 23:15 Mohammad AlSaleh < notifications@github.com> ha scritto:

…

I wish macOS users would start including kitty <https://github.com/kovidgoyal/kitty> when they are reporting their experiences. I'm not a macOS user myself, so I have no idea how performant iTerm2 is. But I tested kitty in a GNU/Linux system, and it is a decent emulator. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#289 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFnUpku-DTfeBuo9LHM9XJwaGJAwoRnsks5t1dFugaJpZM4LgvIl> .

[MoSal]

@lompabo

That's weird. kitty on GNU/Linux tests okay. Definitely not X times slower than the best.

Test from this comment (best out of 5 runs, same font/font-size):

Terminal	Time	Version
lxterminal	30.2	0.3.1
kitty	22.5	0.10.0
konsole	18.6 (cheating)	18.04.1
alacritty	16.7	7c4e84c
st	15.3	0.8.1

[chrisduerr]

The benchmarking procedure of Alacritty is documented in the blog post about the scrollback release:
https://jwilm.io/blog/alacritty-lands-scrollback

There's still work to be done here, like automated benchmarks and including tests for things like terminal latency. But I think this should resolve this issue for now.

If anyone wants to help out with benchmarking Alacritty, the https://github.com/jwilm/vtebench repository is where all the code is present.