New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open-sourced your benchmark? #289

Open
Naereen opened this Issue Jan 11, 2017 · 38 comments

Comments

Projects
None yet
@Naereen

Naereen commented Jan 11, 2017

Hi,

  • First, I just tested your project on XUbuntu 16.10, and everything works fine. Good job, it's easy to install and launch.
  • I think your warning about the config file in the README.md file might not be necessary (as you say, if the config file is not present, it is created on the first run of alacritty).

I tested alacritty's performance against xfce4-terminal, on simple programs, and on cmatrix. My observation is that runs slightly slower than xfce4-terminal.

But it is unclear how you measured the performance and what kind of benchmarks did you use.
Could it be possible to explain more, and maybe open-source whatever software or script was used for the benchmark?

I would be curious to test again, with a more robust measure.
Thanks.

@elopez

This comment has been minimized.

Show comment
Hide comment
@elopez

elopez Jan 12, 2017

I'd like to know how performance was measured too. I ran a non-scientific benchmark: executing "time find /usr" a bunch of times on an Arch Linux machine with an SSD. Alacritty took about 40s to print all of it, while GNOME Terminal always finished in under 3s. Even xterm did better, doing it under 6s.

elopez commented Jan 12, 2017

I'd like to know how performance was measured too. I ran a non-scientific benchmark: executing "time find /usr" a bunch of times on an Arch Linux machine with an SSD. Alacritty took about 40s to print all of it, while GNOME Terminal always finished in under 3s. Even xterm did better, doing it under 6s.

@jwilm

This comment has been minimized.

Show comment
Hide comment
@jwilm

jwilm Jan 12, 2017

Owner

Benchmarks so far have just involved running find /usr on my Linux system with Alacritty, st, and urxvt, and on macOS against Terminal.app and iTerm2.

Owner

jwilm commented Jan 12, 2017

Benchmarks so far have just involved running find /usr on my Linux system with Alacritty, st, and urxvt, and on macOS against Terminal.app and iTerm2.

@alexhultman

This comment has been minimized.

Show comment
Hide comment
@alexhultman

alexhultman Jan 12, 2017

maybe it is not a good idea to claim to be fastest (it is a superlative) when you havent tested more than 3 projects.

alexhultman commented Jan 12, 2017

maybe it is not a good idea to claim to be fastest (it is a superlative) when you havent tested more than 3 projects.

@MoSal

This comment has been minimized.

Show comment
Hide comment
@MoSal

MoSal Jan 12, 2017

Benchmarks so far have just involved running find /usr on my Linux system with Alacritty, st, and urxvt, and on macOS against Terminal.app and iTerm2.

That's hardly an extensive test scenario. And an odd choice of GNU/Linux terminals.

find /usr doesn't cover heavy usage of many codes. Testing with saldl , for example, immediately exposes color issues, flashing every few seconds, among other issues. Those flashes are not seen when using vte-based terminals, konsole, mlterm, or OSX's Terminal. But I do vaguely remember seeing them in ConEmu, and maybe other slow emulators.

While the potential is there, I don't think Alacritty can be called a fast, let alone the fastest, terminal emulator at the moment.

MoSal commented Jan 12, 2017

Benchmarks so far have just involved running find /usr on my Linux system with Alacritty, st, and urxvt, and on macOS against Terminal.app and iTerm2.

That's hardly an extensive test scenario. And an odd choice of GNU/Linux terminals.

find /usr doesn't cover heavy usage of many codes. Testing with saldl , for example, immediately exposes color issues, flashing every few seconds, among other issues. Those flashes are not seen when using vte-based terminals, konsole, mlterm, or OSX's Terminal. But I do vaguely remember seeing them in ConEmu, and maybe other slow emulators.

While the potential is there, I don't think Alacritty can be called a fast, let alone the fastest, terminal emulator at the moment.

@Naereen

This comment has been minimized.

Show comment
Hide comment
@Naereen

Naereen Jan 12, 2017

I agree with @MoSal.

Naereen commented Jan 12, 2017

I agree with @MoSal.

@roobie

This comment has been minimized.

Show comment
Hide comment
@roobie

roobie Jan 12, 2017

Commands one can use to generate output:

  • dd if=/dev/urandom | base64
  • cd /; tree

I tested these with Terminator, Sakura and Alacritty, and Alacritty performed very well

EDIT: Dunno if executing inside a tmux session makes a difference, but I tested that as well, which yielded good results.

roobie commented Jan 12, 2017

Commands one can use to generate output:

  • dd if=/dev/urandom | base64
  • cd /; tree

I tested these with Terminator, Sakura and Alacritty, and Alacritty performed very well

EDIT: Dunno if executing inside a tmux session makes a difference, but I tested that as well, which yielded good results.

@jwilm

This comment has been minimized.

Show comment
Hide comment
@jwilm

jwilm Jan 13, 2017

Owner

I'd like to apologize to those of you who feel mislead about performance claims. You're right that 4 comparisons is not particularly thorough given the volume of terminal emulators on the market. FWIW, I've been doing these comparisons all through development and found Alacritty to be the fastest.

Moving forward, I plan to continue marketing Alacritty as the fastest terminal emulator available and back that up with indisputable benchmarks. It seems like there's a bit of work to do on that front.

while already at it, isnt every terminal emulator "GPU accelerated"? i find it very hard to believe gnome terminal, as an example wouldnt use the graphics card at all. seems kind of strange to make a big deal out of utilizing the graphics card to draw graphics in 2017 - but maybe im wrong?

Assuming for a moment that X drawing APIs use the GPU wherever possible, drawing APIs like that are nearly always done in immediate mode. This mode is incredibly wasteful in the context of a GPU due to frequent uploads, lack of batching, and frequent state changes.

Contrast this with what Alacritty does--retained mode rendering. Display lists for what's on the screen are precomputed from the terminal state. Those lists are uploaded to the GPU, and then everything is drawn at once using only two draw commands.

Owner

jwilm commented Jan 13, 2017

I'd like to apologize to those of you who feel mislead about performance claims. You're right that 4 comparisons is not particularly thorough given the volume of terminal emulators on the market. FWIW, I've been doing these comparisons all through development and found Alacritty to be the fastest.

Moving forward, I plan to continue marketing Alacritty as the fastest terminal emulator available and back that up with indisputable benchmarks. It seems like there's a bit of work to do on that front.

while already at it, isnt every terminal emulator "GPU accelerated"? i find it very hard to believe gnome terminal, as an example wouldnt use the graphics card at all. seems kind of strange to make a big deal out of utilizing the graphics card to draw graphics in 2017 - but maybe im wrong?

Assuming for a moment that X drawing APIs use the GPU wherever possible, drawing APIs like that are nearly always done in immediate mode. This mode is incredibly wasteful in the context of a GPU due to frequent uploads, lack of batching, and frequent state changes.

Contrast this with what Alacritty does--retained mode rendering. Display lists for what's on the screen are precomputed from the terminal state. Those lists are uploaded to the GPU, and then everything is drawn at once using only two draw commands.

@Naereen

This comment has been minimized.

Show comment
Hide comment
@Naereen

Naereen Jan 13, 2017

Moving forward, I plan to continue marketing Alacritty as the fastest terminal emulator available

It's good to stay positive and motivated about your project, although I really think a serious and complete benchmark is needed (e.g., with plots and tabs of time values, for several heavy commands, as well as CPU/GPU/RAM usage for more than 10 emulators).

One reason for asking for more than find / or cd / ; tree examples is that these examples do not use ANSI color codes, do not use non-ASCII symbols or glyphs.

See for instance the benchmarks displayed on Julia's homepage.

Naereen commented Jan 13, 2017

Moving forward, I plan to continue marketing Alacritty as the fastest terminal emulator available

It's good to stay positive and motivated about your project, although I really think a serious and complete benchmark is needed (e.g., with plots and tabs of time values, for several heavy commands, as well as CPU/GPU/RAM usage for more than 10 emulators).

One reason for asking for more than find / or cd / ; tree examples is that these examples do not use ANSI color codes, do not use non-ASCII symbols or glyphs.

See for instance the benchmarks displayed on Julia's homepage.

@Tsutsukakushi

This comment has been minimized.

Show comment
Hide comment
@Tsutsukakushi

Tsutsukakushi Jan 13, 2017

@Naereen actually tree will use LS_COLORS if it's set.

Also don't forget about caches when benchmarking. Consider the following script where alacritty is sure to beat urxvt.

#!/bin/bash
time urxvt -e find /
time alacritty -e find /

A more fair benchmark would be:

#!/bin/bash
time urxvt -e find /
echo 3 > /proc/sys/vm/drop_caches
time alacritty -e find /

Tsutsukakushi commented Jan 13, 2017

@Naereen actually tree will use LS_COLORS if it's set.

Also don't forget about caches when benchmarking. Consider the following script where alacritty is sure to beat urxvt.

#!/bin/bash
time urxvt -e find /
time alacritty -e find /

A more fair benchmark would be:

#!/bin/bash
time urxvt -e find /
echo 3 > /proc/sys/vm/drop_caches
time alacritty -e find /
@lukaslueg

This comment has been minimized.

Show comment
Hide comment
@lukaslueg

lukaslueg Jan 13, 2017

Contributor

We simply need some real #[benchmark]s in alacritty. They can include synthetic tests ("parse 100mb of random stuff", "draw the parsed output of 100mb of random stuff offscreen") and cooked up tests ("here is what the usual output on 12 panes of colored Vim does to you!"). It's the only way to tell if performance increases or decreases over time.

Contributor

lukaslueg commented Jan 13, 2017

We simply need some real #[benchmark]s in alacritty. They can include synthetic tests ("parse 100mb of random stuff", "draw the parsed output of 100mb of random stuff offscreen") and cooked up tests ("here is what the usual output on 12 panes of colored Vim does to you!"). It's the only way to tell if performance increases or decreases over time.

@theduke

This comment has been minimized.

Show comment
Hide comment
@theduke

theduke Jan 13, 2017

For reference, I'm posting my results here, including a good method of generating random data.

# Generate 100MB of random tex.
cat /dev/urandom | base64 | dd of=/tmp/randomdata bs=1024 count=100KB

# Test. Repeat for each terminal...
# Results are written to bench_XXX.txt
# This also shows MB/s very nicely thanks to `dd`.
{ time dd if=/tmp/randomdata bs=10240; } 2> bench_XXX.txt

My results, on Linux Arch with proprietary Nvidia drivers:

  • Terminator: 4.4 MB/s
  • urxvt: 19.3 MB/s
  • alacritty: 24.5 MB/s

theduke commented Jan 13, 2017

For reference, I'm posting my results here, including a good method of generating random data.

# Generate 100MB of random tex.
cat /dev/urandom | base64 | dd of=/tmp/randomdata bs=1024 count=100KB

# Test. Repeat for each terminal...
# Results are written to bench_XXX.txt
# This also shows MB/s very nicely thanks to `dd`.
{ time dd if=/tmp/randomdata bs=10240; } 2> bench_XXX.txt

My results, on Linux Arch with proprietary Nvidia drivers:

  • Terminator: 4.4 MB/s
  • urxvt: 19.3 MB/s
  • alacritty: 24.5 MB/s
@MoSal

This comment has been minimized.

Show comment
Hide comment
@MoSal

MoSal Jan 13, 2017

Here is a more useful test:

while true; do
  sleep 0.01
  echo -e '\r'
  echo -e '\033[0K\033[1mBold\033[0m \033[7mInvert\033[0m \033[4mUnderline\033[0m'
  echo -e '\033[0K\033[1m\033[7m\033[4mBold & Invert & Underline\033[0m'
  echo
  echo -e '\033[0K\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[30m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[30m\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A'
done

Reduce the sleep interval until the cursor movement becomes obvious. Or remove the sleep, and the last echo, and repeat for X times.

I'd suspect the raw performance will greatly vary depending on the GPU/CPU combination.

For me, the major issue is the flashing/blinking, which I'm seeing every few seconds in alacritty, even with long sleep intervals.

(Off-topic: With zero and positive y offsets. Text seems to be bottom-adjusted instead of centered in each line. Is the default negative to work around this known issue?)

MoSal commented Jan 13, 2017

Here is a more useful test:

while true; do
  sleep 0.01
  echo -e '\r'
  echo -e '\033[0K\033[1mBold\033[0m \033[7mInvert\033[0m \033[4mUnderline\033[0m'
  echo -e '\033[0K\033[1m\033[7m\033[4mBold & Invert & Underline\033[0m'
  echo
  echo -e '\033[0K\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[30m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[30m\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A\033[1A'
done

Reduce the sleep interval until the cursor movement becomes obvious. Or remove the sleep, and the last echo, and repeat for X times.

I'd suspect the raw performance will greatly vary depending on the GPU/CPU combination.

For me, the major issue is the flashing/blinking, which I'm seeing every few seconds in alacritty, even with long sleep intervals.

(Off-topic: With zero and positive y offsets. Text seems to be bottom-adjusted instead of centered in each line. Is the default negative to work around this known issue?)

@quininer

This comment has been minimized.

Show comment
Hide comment
@quininer

quininer May 10, 2017

Contributor

I have an intuitive way to test terminal performance.
use mpv --vo tct to play a video and see if it is dropped frame.

Contributor

quininer commented May 10, 2017

I have an intuitive way to test terminal performance.
use mpv --vo tct to play a video and see if it is dropped frame.

markstos added a commit to markstos/alacritty that referenced this issue Sep 28, 2017

Correct accuracy of first sentence in README
As documented in jwilm#289 it's not currently the fastest-- only a small number of terminals running a single task for initially benchmarked.
@kutsan

This comment has been minimized.

Show comment
Hide comment
@kutsan

kutsan Oct 28, 2017

I agree, claiming that it's the fastest terminal emulator out there, is questionable. Using system binaries like find does not produce consistent result across other systems. There are so much factors that affect its benchmark. Even the version of find used to test. You should frankly and clearly demonstrate why it's the fastest and how. It would be nice to create a platform agnostic benchmark script to test it. So, people can run and compare their results in different terminal emulators and your results.

kutsan commented Oct 28, 2017

I agree, claiming that it's the fastest terminal emulator out there, is questionable. Using system binaries like find does not produce consistent result across other systems. There are so much factors that affect its benchmark. Even the version of find used to test. You should frankly and clearly demonstrate why it's the fastest and how. It would be nice to create a platform agnostic benchmark script to test it. So, people can run and compare their results in different terminal emulators and your results.

@tokudan

This comment has been minimized.

Show comment
Hide comment
@tokudan

tokudan Oct 28, 2017

seq from GNU coreutils generates a simple stream of numbers and is available on most systems:
$ time seq 1 999999

My result with xfce4-terminal:
real 0m13,890s
user 0m0,020s
sys 0m0,542s

My result with alacritty:
real 0m0,524s
user 0m0,016s
sys 0m0,465s

tokudan commented Oct 28, 2017

seq from GNU coreutils generates a simple stream of numbers and is available on most systems:
$ time seq 1 999999

My result with xfce4-terminal:
real 0m13,890s
user 0m0,020s
sys 0m0,542s

My result with alacritty:
real 0m0,524s
user 0m0,016s
sys 0m0,465s

@Naereen

This comment has been minimized.

Show comment
Hide comment
@Naereen

Naereen Oct 28, 2017

@tokudan : one command, on one machine, comparing only two terminal emulators, and ran only once is NOT at all a benchmark...

Naereen commented Oct 28, 2017

@tokudan : one command, on one machine, comparing only two terminal emulators, and ran only once is NOT at all a benchmark...

@kmicu

This comment has been minimized.

Show comment
Hide comment
@kmicu

kmicu Oct 28, 2017

But it is still a fact and still a data point which shows relative speed on some workstation.

Here is more data points (executed multiple times, but results were not consistent only in case of rxvt):

Shell: GNU bash 4.4.12
DDX: modesetting
CPU: i7-4702MQ
GPU: Intel HD Graphics 4600
Compositor: compton-0.1_beta2.5
I do not bother with listing next x things that could affect rendering speed.
Relative speed is important here.

Test command: time for i in {1..3}; do seq 1 999999; done

XTerm 330
visually: flashy and uneven, stroboscopic

real 3m2.195s
user 0m0.092s
sys 0m2.004s

Alacritty
visually: consistent, no problems
real 0m1.377s
user 0m0.075s
sys 0m1.299s

st 0.7
visually: consistent, no problems

real 0m4.116s
user 0m0.067s
sys 0m1.747s

uxrvt v9.22
visually: consistent, no problems

different results on each invocation
from:
real 0m1.081s
user 0m0.052s
sys 0m0.864s

to:
real 0m1.626s
user 0m0.079s
sys 0m0.942s

xfce4-terminal 0.6.3
visually: jumpy prompt

real 0m44.232s
user 0m0.128s
sys 0m3.342s

Creating a proper benchmark would be really difficult if you take into account that hardware AND drivers AND Xorg config can affect results. So at the same time on my workstation alacritty can be the fastest and on your workstation can be the slowest. Good luck with buying hundreds of computers to create a proper, honest, reproducible benchmark to judge if a terminal emulator is really the fastest in the whole universe (づ ̄ ³ ̄)づ

kmicu commented Oct 28, 2017

But it is still a fact and still a data point which shows relative speed on some workstation.

Here is more data points (executed multiple times, but results were not consistent only in case of rxvt):

Shell: GNU bash 4.4.12
DDX: modesetting
CPU: i7-4702MQ
GPU: Intel HD Graphics 4600
Compositor: compton-0.1_beta2.5
I do not bother with listing next x things that could affect rendering speed.
Relative speed is important here.

Test command: time for i in {1..3}; do seq 1 999999; done

XTerm 330
visually: flashy and uneven, stroboscopic

real 3m2.195s
user 0m0.092s
sys 0m2.004s

Alacritty
visually: consistent, no problems
real 0m1.377s
user 0m0.075s
sys 0m1.299s

st 0.7
visually: consistent, no problems

real 0m4.116s
user 0m0.067s
sys 0m1.747s

uxrvt v9.22
visually: consistent, no problems

different results on each invocation
from:
real 0m1.081s
user 0m0.052s
sys 0m0.864s

to:
real 0m1.626s
user 0m0.079s
sys 0m0.942s

xfce4-terminal 0.6.3
visually: jumpy prompt

real 0m44.232s
user 0m0.128s
sys 0m3.342s

Creating a proper benchmark would be really difficult if you take into account that hardware AND drivers AND Xorg config can affect results. So at the same time on my workstation alacritty can be the fastest and on your workstation can be the slowest. Good luck with buying hundreds of computers to create a proper, honest, reproducible benchmark to judge if a terminal emulator is really the fastest in the whole universe (づ ̄ ³ ̄)づ

@alexhultman

This comment has been minimized.

Show comment
Hide comment
@alexhultman

alexhultman Oct 28, 2017

Regarding my first comment about superlatives (in the top), I just looked at the front page and saw this as the very first text:

"Alacritty is the fastest terminal emulator in existence."

I don't know about you, but this makes me throw up in my mouth. First of all, this is socially embarrassing behavior. You cannot possibly know this to be true, and it is awfully boastful. Not only is it implying "in the world" but also "in the entire universe". This, simply is bullshittery x2000.

Secondly, people have reported all kinds of numbers showing that in some cases this simply isn't even true. Since you obviously had a successful initial marketing, maybe it is time to tone down this aggressive marketing a bit. It is tasteless and tacky.

alexhultman commented Oct 28, 2017

Regarding my first comment about superlatives (in the top), I just looked at the front page and saw this as the very first text:

"Alacritty is the fastest terminal emulator in existence."

I don't know about you, but this makes me throw up in my mouth. First of all, this is socially embarrassing behavior. You cannot possibly know this to be true, and it is awfully boastful. Not only is it implying "in the world" but also "in the entire universe". This, simply is bullshittery x2000.

Secondly, people have reported all kinds of numbers showing that in some cases this simply isn't even true. Since you obviously had a successful initial marketing, maybe it is time to tone down this aggressive marketing a bit. It is tasteless and tacky.

@kmicu

This comment has been minimized.

Show comment
Hide comment
@kmicu

kmicu Oct 29, 2017

People here criticizing facts and data have no idea how difficult task is to create a benchmark for rendering and how useful it would be in the end. Your imaginary benchmark is not only affected by what you do in a benchmark but also by its environment. So DDX drivers, hardware, a compositor config, and so on. So even if you get some numbers on your precious benchmark they are meaningless for others if you do not have bit by bit perfectly reproducible hardware and software.

@alexhultman “Secondly, people have reported all kinds of numbers showing that in some cases this simply isn't even true” and at the same time you are ignoring the fact that majority of numbers reported here show that Alacritty has the fastest rendering. You did not provide any additional data point disproving that.

If you folks are so much offended by Alacritty’s claims then show us data and not only emotional comments. On my setup Alacritty is marginally faster than (inconsistent) uxrvt. So I have no reasons to complain. If you do not like proposed seq 1 999999 or mpv --vo tct tests then show us a better one. Do something useful. Any data point is worth more than your complaints.

kmicu commented Oct 29, 2017

People here criticizing facts and data have no idea how difficult task is to create a benchmark for rendering and how useful it would be in the end. Your imaginary benchmark is not only affected by what you do in a benchmark but also by its environment. So DDX drivers, hardware, a compositor config, and so on. So even if you get some numbers on your precious benchmark they are meaningless for others if you do not have bit by bit perfectly reproducible hardware and software.

@alexhultman “Secondly, people have reported all kinds of numbers showing that in some cases this simply isn't even true” and at the same time you are ignoring the fact that majority of numbers reported here show that Alacritty has the fastest rendering. You did not provide any additional data point disproving that.

If you folks are so much offended by Alacritty’s claims then show us data and not only emotional comments. On my setup Alacritty is marginally faster than (inconsistent) uxrvt. So I have no reasons to complain. If you do not like proposed seq 1 999999 or mpv --vo tct tests then show us a better one. Do something useful. Any data point is worth more than your complaints.

@MoSal

This comment has been minimized.

Show comment
Hide comment
@MoSal

MoSal Oct 29, 2017

@kmicu
Can you benchmark alacritty and konsole with this test:

for i in {1..400000}; do
  echo -e '\r'
  echo -e '\033[0K\033[1mBold\033[0m \033[7mInvert\033[0m \033[4mUnderline\033[0m'
  echo -e '\033[0K\033[1m\033[7m\033[4mBold & Invert & Underline\033[0m'
  echo
  echo -e '\033[0K\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[30m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[30m\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
done

MoSal commented Oct 29, 2017

@kmicu
Can you benchmark alacritty and konsole with this test:

for i in {1..400000}; do
  echo -e '\r'
  echo -e '\033[0K\033[1mBold\033[0m \033[7mInvert\033[0m \033[4mUnderline\033[0m'
  echo -e '\033[0K\033[1m\033[7m\033[4mBold & Invert & Underline\033[0m'
  echo
  echo -e '\033[0K\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[31m Red \033[32m Green \033[33m Yellow \033[34m Blue \033[35m Magenta \033[36m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo
  echo -e '\033[0K\033[30m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
  echo -e '\033[0K\033[30m\033[1m\033[4m\033[41m Red \033[42m Green \033[43m Yellow \033[44m Blue \033[45m Magenta \033[46m Cyan \033[0m'
done
@fschutt

This comment has been minimized.

Show comment
Hide comment
@fschutt

fschutt Oct 29, 2017

for @MoSal s benchmark:

  • konsole: real 0m38,987s, user 0m31,216s, sys 0m7,744s
  • alacritty: real 0m40,352s, user 0m32,380s, sys 0m7,956s
  • gnome-terminal: real 1m7,990s, user 0m33,292s, sys 0m9,012s
  • xterm: more than 5 minutes

All terminals had the same size, laptop with SSD, Intel HD drivers on X11. I would not claim that alacritty is "the fastest in all cases", but it's pretty fast. However, we are currently arguing about a 1 second speed difference of terminal emulators in a synthetic benchmark with 400K iterations. There are more important things in life.

fschutt commented Oct 29, 2017

for @MoSal s benchmark:

  • konsole: real 0m38,987s, user 0m31,216s, sys 0m7,744s
  • alacritty: real 0m40,352s, user 0m32,380s, sys 0m7,956s
  • gnome-terminal: real 1m7,990s, user 0m33,292s, sys 0m9,012s
  • xterm: more than 5 minutes

All terminals had the same size, laptop with SSD, Intel HD drivers on X11. I would not claim that alacritty is "the fastest in all cases", but it's pretty fast. However, we are currently arguing about a 1 second speed difference of terminal emulators in a synthetic benchmark with 400K iterations. There are more important things in life.

@MoSal

This comment has been minimized.

Show comment
Hide comment
@MoSal

MoSal Oct 30, 2017

@fschutt

However, we are currently arguing about a 1 second speed difference of terminal emulators in a synthetic benchmark with 400K iterations. There are more important things in life.

Fully agree*. I just provided this counter test-case because someone asked for one.

The issue I'm still seeing with alacritty is slow cursor movement, which would manifest as flashing
in slower hardware (test case in my earlier comment). The issue is less pronounced in my new hardware. But it's still there.

* FWIW, I'm not a KDE user. And I don't use konsole because I think it's the fastest. I use it because it's the only one with usable BiDi support (less smart, but works better than mlterm).

MoSal commented Oct 30, 2017

@fschutt

However, we are currently arguing about a 1 second speed difference of terminal emulators in a synthetic benchmark with 400K iterations. There are more important things in life.

Fully agree*. I just provided this counter test-case because someone asked for one.

The issue I'm still seeing with alacritty is slow cursor movement, which would manifest as flashing
in slower hardware (test case in my earlier comment). The issue is less pronounced in my new hardware. But it's still there.

* FWIW, I'm not a KDE user. And I don't use konsole because I think it's the fastest. I use it because it's the only one with usable BiDi support (less smart, but works better than mlterm).

@kmicu

This comment has been minimized.

Show comment
Hide comment
@kmicu

kmicu Oct 30, 2017

@MoSal Here we go:

(the same workstation and software/config versions as in my previous post)

visually: consistent, no jumping
real 0m27.552s
user 0m21.436s
sys 0m6.115s
real 0m28.232s
user 0m21.887s
sys 0m6.344s
real 0m27.728s
user 0m21.480s
sys 0m6.244s

Konsole 17.08.1
visually: jumpy and uneven, it goes, stops for a moment and goes again and so on
real 0m28.957s
user 0m20.540s
sys 0m5.671s
real 0m29.755s
user 0m20.441s
sys 0m5.811s
real 0m29.838s
user 0m21.428s
sys 0m5.751s

st: a little bit jumpy, but ok
real 0m26.933s
user 0m20.946s
sys 0m5.927s
real 0m26.575s
user 0m20.726s
sys 0m5.826s
real 0m26.868s
user 0m21.003s
sys 0m5.819s

rxvt
visually: a little bit jumpy, but ok
real 0m30.615s
user 0m21.345s
sys 0m6.231s
real 0m31.122s
user 0m21.478s
sys 0m6.032s
real 0m31.437s
user 0m21.875s
sys 0m6.211s

I do not like this particular test because 100% CPU is taken also by a shell process (bash, zsh, etc). So we are not testing only rendering, but also some funky interaction with a shell.

PS
Please, keep in mind that above numbers apply only to my workstation: a custom Compton config, DDX modesetting, a custom Xorg 1.19.5 config, EGL version 1.4 (DRI2) and so on.
Fun fact: I use UXTerm as my daily driver, because of http://invisible-island.net/xterm/xterm.faq.html#compare_versions

kmicu commented Oct 30, 2017

@MoSal Here we go:

(the same workstation and software/config versions as in my previous post)

visually: consistent, no jumping
real 0m27.552s
user 0m21.436s
sys 0m6.115s
real 0m28.232s
user 0m21.887s
sys 0m6.344s
real 0m27.728s
user 0m21.480s
sys 0m6.244s

Konsole 17.08.1
visually: jumpy and uneven, it goes, stops for a moment and goes again and so on
real 0m28.957s
user 0m20.540s
sys 0m5.671s
real 0m29.755s
user 0m20.441s
sys 0m5.811s
real 0m29.838s
user 0m21.428s
sys 0m5.751s

st: a little bit jumpy, but ok
real 0m26.933s
user 0m20.946s
sys 0m5.927s
real 0m26.575s
user 0m20.726s
sys 0m5.826s
real 0m26.868s
user 0m21.003s
sys 0m5.819s

rxvt
visually: a little bit jumpy, but ok
real 0m30.615s
user 0m21.345s
sys 0m6.231s
real 0m31.122s
user 0m21.478s
sys 0m6.032s
real 0m31.437s
user 0m21.875s
sys 0m6.211s

I do not like this particular test because 100% CPU is taken also by a shell process (bash, zsh, etc). So we are not testing only rendering, but also some funky interaction with a shell.

PS
Please, keep in mind that above numbers apply only to my workstation: a custom Compton config, DDX modesetting, a custom Xorg 1.19.5 config, EGL version 1.4 (DRI2) and so on.
Fun fact: I use UXTerm as my daily driver, because of http://invisible-island.net/xterm/xterm.faq.html#compare_versions

@alexanderadam

This comment has been minimized.

Show comment
Hide comment
@alexanderadam

alexanderadam Oct 31, 2017

Wow, @kmicu your PS is gold. It would be nice to have this compatibility for alacritty as well.

alexanderadam commented Oct 31, 2017

Wow, @kmicu your PS is gold. It would be nice to have this compatibility for alacritty as well.

@AlexisTM

This comment has been minimized.

Show comment
Hide comment
@AlexisTM

AlexisTM Jan 16, 2018

Tmux performance issue? What am I doing wrong?

$ time seq 1 999999

[...]

real    0m0.444s
user    0m0.025s
sys     0m0.419s

$ tmux

$ echo $TERM
alacritty

$ time seq 1 999999

[...]

real    0m5.444s
user    0m0.042s
sys     0m0.562s

AlexisTM commented Jan 16, 2018

Tmux performance issue? What am I doing wrong?

$ time seq 1 999999

[...]

real    0m0.444s
user    0m0.025s
sys     0m0.419s

$ tmux

$ echo $TERM
alacritty

$ time seq 1 999999

[...]

real    0m5.444s
user    0m0.042s
sys     0m0.562s
@Swoorup

This comment has been minimized.

Show comment
Hide comment
@Swoorup

Swoorup Apr 19, 2018

This is bit worrying. I used alacritty because of claims of being the fastest and in rust. But it doesn't turn out to be the reality as the tests shown here.

Swoorup commented Apr 19, 2018

This is bit worrying. I used alacritty because of claims of being the fastest and in rust. But it doesn't turn out to be the reality as the tests shown here.

@chrisduerr

This comment has been minimized.

Show comment
Hide comment
@chrisduerr

chrisduerr Apr 19, 2018

Collaborator

@Swoorup Make sure you use the scrollback PR when testing. In my tests alacritty is the fastest terminal emulator in all current benchmarks (except for unicode stuff but that's not really related to "raw speed"). Even on master the random write is significantly faster though, just the scrolling had some performance tweaks.

Collaborator

chrisduerr commented Apr 19, 2018

@Swoorup Make sure you use the scrollback PR when testing. In my tests alacritty is the fastest terminal emulator in all current benchmarks (except for unicode stuff but that's not really related to "raw speed"). Even on master the random write is significantly faster though, just the scrolling had some performance tweaks.

@vks

This comment has been minimized.

Show comment
Hide comment
@vks

vks Apr 23, 2018

Terminal latency should probably be considered as well, see for example these benchmarks: https://danluu.com/term-latency/

vks commented Apr 23, 2018

Terminal latency should probably be considered as well, see for example these benchmarks: https://danluu.com/term-latency/

@fschutt

This comment has been minimized.

Show comment
Hide comment
@fschutt

fschutt Apr 25, 2018

In https://gitlab.com/anarcat/terms-benchmarks, alacritty is fairly good and if you'd trust that benchmark, then yes, alacritty can claim that it is the fastest terminal emulator. Also see the article on LWN: https://lwn.net/Articles/749992/ - I think it depends on which benchmark you believe and in the end, everyone has to decide for itself.

In some benchmarks, alacritty is truly the fastest emulator, in others it isn't. I mean, since alacritty is built from source, there isn't even a guarantee that we are all testing the same binary. I do think that "alacritty is the fastest" is a lie, since it's not the fastest in all cases and in all benchmarks. It's also not the fastest by a huge margin, so differences of a few milliseconds can be attributed to system hiccups, other processes running in the background, etc. Some emulators cheat by not updating the whole screen at once, etc. Benchmarking is hard. However it's "one of the fastest", so I'll let it slide, if it motivates him to continue working on it and making alacritty faster.

fschutt commented Apr 25, 2018

In https://gitlab.com/anarcat/terms-benchmarks, alacritty is fairly good and if you'd trust that benchmark, then yes, alacritty can claim that it is the fastest terminal emulator. Also see the article on LWN: https://lwn.net/Articles/749992/ - I think it depends on which benchmark you believe and in the end, everyone has to decide for itself.

In some benchmarks, alacritty is truly the fastest emulator, in others it isn't. I mean, since alacritty is built from source, there isn't even a guarantee that we are all testing the same binary. I do think that "alacritty is the fastest" is a lie, since it's not the fastest in all cases and in all benchmarks. It's also not the fastest by a huge margin, so differences of a few milliseconds can be attributed to system hiccups, other processes running in the background, etc. Some emulators cheat by not updating the whole screen at once, etc. Benchmarking is hard. However it's "one of the fastest", so I'll let it slide, if it motivates him to continue working on it and making alacritty faster.

@chrisduerr

This comment has been minimized.

Show comment
Hide comment
@chrisduerr

chrisduerr Apr 25, 2018

Collaborator

You should also make sure that whenever benchmarking things like scrolling (yes for example) to make use of the scrollback PR, there were some significant speed enhancements made in that PR which lead to alacritty taking the lead over both urxvt and st on my machine.

Regarding random updates (so raw terminal throughput) alacritty seems to beat the competition by a fair margin.

Collaborator

chrisduerr commented Apr 25, 2018

You should also make sure that whenever benchmarking things like scrolling (yes for example) to make use of the scrollback PR, there were some significant speed enhancements made in that PR which lead to alacritty taking the lead over both urxvt and st on my machine.

Regarding random updates (so raw terminal throughput) alacritty seems to beat the competition by a fair margin.

@therealraven

This comment has been minimized.

Show comment
Hide comment
@therealraven

therealraven May 23, 2018

Alacritty makes the terminal experience significantly better than iTerm2 on Mac OS. Many people know this. Please don't let negative comments in threads like this one discourage you. The terminal you have built is indispensable to Mac OS users.

Benchmarks aside for a moment, on Mac OS simply scrolling in Vim in Tmux in Alacritty is much faster in Alacritty compared to Terminal.app or iTerm2. It has a real impact on common daily tasks.

therealraven commented May 23, 2018

Alacritty makes the terminal experience significantly better than iTerm2 on Mac OS. Many people know this. Please don't let negative comments in threads like this one discourage you. The terminal you have built is indispensable to Mac OS users.

Benchmarks aside for a moment, on Mac OS simply scrolling in Vim in Tmux in Alacritty is much faster in Alacritty compared to Terminal.app or iTerm2. It has a real impact on common daily tasks.

@ngocphamm

This comment has been minimized.

Show comment
Hide comment
@ngocphamm

ngocphamm May 23, 2018

I don't know why but scrolling (holding down j/k) in Neovim, in Tmux using Alacritty and iTerm2 (3.1 or 3.2 beta now) doesn't feel much different to me, even with rendering using Metal OFF in iTerm2. Terminal.app does feel a little more sluggish, though.

Background: I have KeyRepeat set at 15ms.

ngocphamm commented May 23, 2018

I don't know why but scrolling (holding down j/k) in Neovim, in Tmux using Alacritty and iTerm2 (3.1 or 3.2 beta now) doesn't feel much different to me, even with rendering using Metal OFF in iTerm2. Terminal.app does feel a little more sluggish, though.

Background: I have KeyRepeat set at 15ms.

@chrisduerr

This comment has been minimized.

Show comment
Hide comment
@chrisduerr

chrisduerr May 23, 2018

Collaborator

@therealraven I appreciate the support, I really do :), but at the same time, complaining about alacritty being slow is also support (when done in a respectable manner). Alacritty aims to be blazingly fast, so whenever it's not, that's a bug and reporting such a bug is a good thing. It's unfortunately impossible to test alacritty on every system so it can be kinda tough to troubleshoot and fix some issues without people complaining.

It's always great to hear positive comments, but I don't think (or at least I hope so) anyone is discouraged by valid criticism about shortcomings and bugs about alacritty, so no worries there. :)

Thanks for the feedback!
(Note: I'm just speaking for myself here, but I doubt jwilm or anyone of the other lovely contributors strongly disagrees with any points I've made)

Collaborator

chrisduerr commented May 23, 2018

@therealraven I appreciate the support, I really do :), but at the same time, complaining about alacritty being slow is also support (when done in a respectable manner). Alacritty aims to be blazingly fast, so whenever it's not, that's a bug and reporting such a bug is a good thing. It's unfortunately impossible to test alacritty on every system so it can be kinda tough to troubleshoot and fix some issues without people complaining.

It's always great to hear positive comments, but I don't think (or at least I hope so) anyone is discouraged by valid criticism about shortcomings and bugs about alacritty, so no worries there. :)

Thanks for the feedback!
(Note: I'm just speaking for myself here, but I doubt jwilm or anyone of the other lovely contributors strongly disagrees with any points I've made)

@sandersantema

This comment has been minimized.

Show comment
Hide comment
@sandersantema

sandersantema May 23, 2018

In my experience alacritty is much faster than iTerm and certainly much faster than Terminal.app. Even if Alacritty wouldn't be faster than iTerm i'd still use it since:

  • It's simple, which is a plus for me.
  • It seems stable (even though I'm using a experimental branch)
  • It's the only app which doesn't have those awful borders around the text which I despise so very much, together with the speed of Alacritty this was the biggest reason I've chosen to use Allacritty.

Alacritty is a fantastic application and I believe it only can become better 😄

sandersantema commented May 23, 2018

In my experience alacritty is much faster than iTerm and certainly much faster than Terminal.app. Even if Alacritty wouldn't be faster than iTerm i'd still use it since:

  • It's simple, which is a plus for me.
  • It seems stable (even though I'm using a experimental branch)
  • It's the only app which doesn't have those awful borders around the text which I despise so very much, together with the speed of Alacritty this was the biggest reason I've chosen to use Allacritty.

Alacritty is a fantastic application and I believe it only can become better 😄

@MoSal

This comment has been minimized.

Show comment
Hide comment
@MoSal

MoSal May 23, 2018

I wish macOS users would start including kitty when they are reporting their experiences.

I'm not a macOS user myself, so I have no idea how performant iTerm2 is. But I tested kitty in a GNU/Linux system, and it is a decent emulator.

MoSal commented May 23, 2018

I wish macOS users would start including kitty when they are reporting their experiences.

I'm not a macOS user myself, so I have no idea how performant iTerm2 is. But I tested kitty in a GNU/Linux system, and it is a decent emulator.

@lompabo

This comment has been minimized.

Show comment
Hide comment
@lompabo

lompabo May 24, 2018

lompabo commented May 24, 2018

@MoSal

This comment has been minimized.

Show comment
Hide comment
@MoSal

MoSal May 24, 2018

@lompabo

That's weird. kitty on GNU/Linux tests okay. Definitely not X times slower than the best.

Test from this comment (best out of 5 runs, same font/font-size):

Terminal Time Version
lxterminal 30.2 0.3.1
kitty 22.5 0.10.0
konsole 18.6 (cheating) 18.04.1
alacritty 16.7 7c4e84c
st 15.3 0.8.1

MoSal commented May 24, 2018

@lompabo

That's weird. kitty on GNU/Linux tests okay. Definitely not X times slower than the best.

Test from this comment (best out of 5 runs, same font/font-size):

Terminal Time Version
lxterminal 30.2 0.3.1
kitty 22.5 0.10.0
konsole 18.6 (cheating) 18.04.1
alacritty 16.7 7c4e84c
st 15.3 0.8.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment