tail -f | grep | bat fails (on Linux)

[nekr0z]

When working with logs, it's common to tail -f a log to view it in realtime, and then pipe it to grep for interesting data.

bat fails to produce output when feed such a pipe.

Steps to reproduce the bug

1. Have a long enough log with a common pattern appearing from time to time. /var/log/apt/history.log of a Debian system is used in this example, but any log will do.
2. Find a pattern and allow enough lines for tail to have it: tail -n 1000 -f /var/log/apt/history.log | grep Date should give several lines of output.
3. Pipe the above output to bat, i.e.: tail -n 1000 -f /var/log/apt/history.log | grep Date | bat --paging=never.

Result
The output is empty.

Expected result
The output contains the lines observed on step 2.

More investigation

• If no -f option is given to tail, bat works as expected. (tail -n 1000 /var/log/apt/history.log | grep Date | bat --paging=never produces the expected output)
• If no grep is done, bat works as expected. (tail -n 1000 -f /var/log/apt/history.log | bat --paging=never produces the expected output)

How did you install bat?
Official deb repo.

---

bat version and environment

Software version

bat 0.21.0 (405e5f7)

Operating system

Linux 5.10.0-14-amd64

Command-line

bat --diagnostic 

Environment variables

SHELL=/bin/bash
PAGER=<not set>
LESS=<not set>
LANG=ru_RU.UTF-8
LC_ALL=<not set>
BAT_PAGER=<not set>
BAT_CACHE_PATH=<not set>
BAT_CONFIG_PATH=<not set>
BAT_OPTS=<not set>
BAT_STYLE=<not set>
BAT_TABS=<not set>
BAT_THEME=<not set>
XDG_CONFIG_HOME=<not set>
XDG_CACHE_HOME=<not set>
COLORTERM=truecolor
NO_COLOR=<not set>
MANPAGER=<not set>

Config file

Could not read contents of '/home/evgeny/.config/bat/config': No such file or directory (os error 2).

Custom assets metadata

Could not read contents of '/home/evgeny/.cache/bat/metadata.yaml': No such file or directory (os error 2).

Custom assets

'/home/evgeny/.cache/bat' not found

Compile time information

• Profile: release
• Target triple: x86_64-unknown-linux-gnu
• Family: unix
• OS: linux
• Architecture: x86_64
• Pointer width: 64
• Endian: little
• CPU features: fxsr,sse,sse2
• Host: x86_64-unknown-linux-gnu

Less version

> less --version 
less 551 (GNU regular expressions)
Copyright (C) 1984-2019  Mark Nudelman

less comes with NO WARRANTY, to the extent permitted by law.
For information about the terms of redistribution,
see the file named README in the less distribution.
Home page: http://www.greenwoodsoftware.com/less

[ajgringo619]

What version of tail are you using? I ran your command on (2) different apt-based systems - Debian 11 and Ubuntu 20 - and both produced nothing. However, running the same command on my EndeavourOS system - using tail -n 1000 -f /var/log/pacman.log | grep ALPM-SCRIPTLET | bat --paging=never just to guarantee data - it worked as expected. Since all of my systems use cargo to build bat, that version is the same.

The difference seems to be with tail. EOS uses version 9.1, the apt-based systems are using versions 8.x.

Just to try another set of systems with differing versions of tail, I ran the same test with openSUSE LEAP and openSUSE Tumbleweed...and of course both commands worked flawlessly, even with openSUSE LEAP's 8.3 version of tail. So tail doesn't seem to be the culprit. So I checked the versions of less and found this:

1. All systems that failed have less version 551 or older
2. Systems that succeeded have less version 590

I purposely left all of my testing intact, even though the tail tests seem invalid. Hope you make some headway with this.

[keith-hall]

With bat --paging=never still producing no output, that is an indication that something is wrong outside of bat, because in this case bat is not calling the pager/less. And this can be demonstrated by replacing the bat --paging=never call with cat:

✦ ❯ tail -n 1000 -f /var/log/apt/history.log | grep Date
Start-Date: 2022-08-12  07:50:41
End-Date: 2022-08-12  07:50:44
Start-Date: 2022-08-12  07:50:51
End-Date: 2022-08-12  07:51:01
Start-Date: 2022-08-12  08:01:30
End-Date: 2022-08-12  08:04:45
^C

✦ ❯ tail -n 1000 -f /var/log/apt/history.log | grep Date | cat

🎉 after some investigation, it seems that grep can be told to flush the buffer after each line of output:

✦ ❯ tail -n 1000 -f /var/log/apt/history.log | grep Date --line-buffered | cat
Start-Date: 2022-08-12  07:50:41
End-Date: 2022-08-12  07:50:44
Start-Date: 2022-08-12  07:50:51
End-Date: 2022-08-12  07:51:01
Start-Date: 2022-08-12  08:01:30
End-Date: 2022-08-12  08:04:45
^C