Render ANSI terminal color codes in log buffer

[philomates]

Hello, I was pointed to conjure the other day and am really excited about the prospects of having a nvim clojure plugin mostly written in clojure itself, so really nice work on this!

I noticed that when I use conjure to run tests, ANSI color codes aren't rendered in the log buffer. This is probably due to the fact that a normal text buffer is used for the logs. Using https://github.com/chrisbra/Colorizer I was able to render the colors. I tried a bit to automatically toggle this colorizer plugin on the results of test runs, but couldn't figure it out after a few attempts.

I mainly use iron.nvim + trex.nvim to run clojure tests. This setup uses nvim's terminal feature, which handles ANSI color codes. Investigating a bit how nvim terminal buffers work, I couldn't see how it could be used for this setup.

I'm opening this issue to share my attempt at fixing this and hear if you have any ideas to on how to address this. Feel free to close it if it isn't a main goal of the plugin.

Cheers

[Olical]

Hello, I was pointed to conjure the other day and am really excited about the prospects of having a nvim clojure plugin mostly written in clojure itself, so really nice work on this!

You're more than welcome and I'm glad you're excited about the idea! I hope it ends up being a good tool you enjoy using 😄

I'm open to somehow working around ANSI codes (not fully sure on how yet 🤔) but I'm confused as to how you actually caused some? Like if I cause errors I never see any colour codes... is it a specific library that's doing this? Maybe they just need stripping out for now... for a start?

[philomates]

I think that for stacktraces you don't see ansi codes because the actual data structure is printed and hence normal vim syntax highlighting is applied to it.
In my case I'm using matcher-combinators, which is a testing lib that uses color to help signal mismatches between expected and actual data. You can turn off the ANSI code output in the library itself, but that makes reading test failures a little more difficult.

[Olical]

https://github.com/norcalli/nvim-terminal.lua 👀

Haven't forgotten about this issue, still considering it occasionally.

[daveyarwood]

We have an integration testing project at Adzerk that prints to stdout with ANSI color codes. I've been itching to see if I could get colors working in the Conjure log buffer, so I gave norcalli/nvim-terminal.lua a try. It works, sort of!

The downside is that you activate it by setting filetype to terminal, which overrides the clojure filetype of the Conjure log buffer and removes Clojure syntax highlighting as a side effect. Clojure filetype functionality otherwise seems to be intact, interestingly enough; parinfer and vim-sexp features were still working.

[daveyarwood]

As an additional data point, I tried chrisbra/Colorizer, and it works (without changing the filetype, which is an improvement!), except that it doesn't interpret the "bold" ANSI codes correctly (compare with the screenshot above)

[daveyarwood]

chrisbra/Colorizer is actually turning out to be sort of flaky; I'm not sure that I would 100% recommend it for Conjure purposes at this point.

What I'm seeing is that it will randomly flip back and forth between rendering colors and not rendering colors in the Conjure log buffer, e.g. when I give focus or don't give focus to the Conjure log buffer.

[Olical]

Had a look at this? https://github.com/norcalli/nvim-colorizer.lua

It might not do term codes just yet, but a fork of it embedded into the new Lua based Conjure could solve this issue really easily 🤔

[Olical]

norcalli/nvim-colorizer.lua#31 👀 exciting. I really like this plugin, I use it all the time anyway so it could be good to build a subset into Conjure's log or just recommend it as a pairing once it has term code support?

It's one of the plugins that got me really interested in full Lua plugin development for Neovim actually!

[daveyarwood]

That plugin looks super awesome, but unfortunately, I can't use it because (to my understanding) Ubuntu 16.04 doesn't package Luajit or something. I'm not too familiar with the Lua ecosystem. Perhaps there is a way to install Luajit? The last time I went down this path, it was not a good experience, so I think I'm inclined to just wait for my distro to catch up.

[daveyarwood]

Relevant: norcalli/nvim-colorizer.lua#30

[Olical]

Neovim has Lua support built in :D that's what the new Conjure runs on top of. Try :lua

[daveyarwood]

The issue is about Luajit support, which I think might depend on your OS's Lua package. I apparently don't have Luajit support. I'm not sure if I'm doing it right or wrong here, but:

$ lua
Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
> require 'ffi'
stdin:1: module 'ffi' not found:
        no field package.preload['ffi']
        no file '/home/dave/.lenv/current/luarocks/share/ffi.lua'
        no file '/home/dave/.lenv/current/luarocks/share/ffi/init.lua'
        no file '/usr/local/share/lua/5.3/ffi.lua'
        no file '/usr/local/share/lua/5.3/ffi/init.lua'
        no file '/usr/local/lib/lua/5.3/ffi.lua'
        no file '/usr/local/lib/lua/5.3/ffi/init.lua'
        no file './ffi.lua'
        no file './ffi/init.lua'
        no file '/home/dave/.lenv/current/luarocks/lib/ffi.so'
        no file '/usr/local/lib/lua/5.3/ffi.so'
        no file '/usr/local/lib/lua/5.3/loadall.so'
        no file './ffi.so'
stack traceback:
        [C]: in function 'require'
        stdin:1: in main chunk
        [C]: in ?

[daveyarwood]

It looks like the last time I was trying to get this to work, I ended up installing lenv to try and install a different version of Lua, which may or may not have worked out.

[Olical]

I mean the luajit that is literally compiled into Neovim, give :lua print(10 + 10) a go, that's what that colorizer runs on I think? It doesn't require any Lua interpreter installed on your system. I run it in a tiny Arch Docker container for my tests in CircleCI, there's no luajit installed there.

[Olical]

Or are you saying you tried :lua in Neovim and it exploded? If so, that's bad news for my plans 😬

[Olical]

Oh I see! That lib needs ffi which requires luajit to be installed. I don't think Conjure's rewrite needs anything outside of plain built in lua so hopefully it'll be okay!

[daveyarwood]

:lua print(10 + 10) works and returns 20.

:lua require 'ffi' throws this error:

E5108: Error executing lua [string ":lua"]:1: module 'ffi' not found:
        no field package.preload['ffi']
        no file '/home/dave/.config/nvim/lua/ffi.lua'
        no file '/home/dave/.config/nvim/lua/ffi/init.lua'
... many more lines where it's saying it couldn't find either of those two files in each of my plugin directories ...

I'd say if you don't use anything from Luajit, you're probably fine.

[Olical]

Got it! Thanks for the warning! I was worried I'd misunderstood Neovim's Lua support completely. I think a highlighting module in Conjure's new version could be good though. I could take inspiration from that plugin without using ffi. This idea is still on the cards! So excited for my upcoming time off so I can give it some full time attention. Not being paid for a few months will be... fun? But I'm excited about the possibilities!

…

On Thu, 16 Jan 2020, 17:28 Dave Yarwood, ***@***.***> wrote: :lua print(10 + 10) works and returns 20. :lua require 'ffi' throws this error: E5108: Error executing lua [string ":lua"]:1: module 'ffi' not found: no field package.preload['ffi'] no file '/home/dave/.config/nvim/lua/ffi.lua' no file '/home/dave/.config/nvim/lua/ffi/init.lua' ... many more lines where it's saying it couldn't find either of those two files in each of my plugin directories ... I'd say if you don't use anything from Luajit, you're probably fine. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#53?email_source=notifications&email_token=AACM6XO767FXWTUEKFQYZ5TQ6CKK7A5CNFSM4IJF3N2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJE4B5Y#issuecomment-575258871>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACM6XJI56ONVWONXZ2I3ADQ6CKK7ANCNFSM4IJF3N2A> .

[daveyarwood]

A built-in highlighting module sounds awesome! I think you would only need to support the ANSI color codes, which would simplify things.

[Olical]

So I was just having a look into this and it's definitely harder than I first though, it won't be a quick win since it'll require a LOT of code to handle all of the cases. I'm almost certain I'll miss some and we'll end up with some working and some not. I did stumble across AnsiEsc though which looks amazing, it's implemented exactly how I was planning on doing it! The main problem is that it swaps the highlighting away from whatever you're currently on.

I've been trying to hack it to get it to apply on top of your current colour scheme but I've made no progress yet (Olical/AnsiEsc). Instead I've added the log.strip-ansi-escape-sequences-line-limit option which defaults to 100 lines.

It'll strip ANSI escape sequences from ALL log content less than 100 lines in length, this is so that if you accidentally dump a 100k line output into your log it won't have a series of regex find and replaces run on each line, which should keep it semi-fast. You can turn this off and preserve the codes by setting it to 0.

[Olical]

Okay, my fork of AnsiEsc (Olical/AnsiEsc) actually works to some extent now. It's a big hack, but it overlays the ANSI escape syntax over whatever you're currently using, it worked really well with Clojure! I even got it to work with Janet although that causes some weirdness with the return value highlighting.

So use :ConjureConfig log.strip-ansi-escape-sequences-line-limit 0 to disable the stripping and grab my fork, then run :AnsiEsc in the log buffer to enable it. Not sure how I could enable that automatically, I think that'll require my autocmd framework, until then you'll have to do it manually but I think that's mostly okay.

[Olical]

Here it is working in the HUD 🎉

[daveyarwood]

@Olical I was able to get both the Conjure log buffer and HUD to colorize automatically with this little autocmd:

  " Automatically enable AnsiEsc (interpret ANSI escape codes) for the Conjure
  " log buffer.
  autocmd BufEnter conjure-log-* AnsiEsc

I'm really digging this!

[Olical]

I'm going to finally close this since I think we've got a solution even if it's not the best solution. Being able to turn the ANSI code filtering on or off + a solution for merging ANSI code concealing and syntax into your current syntax were the main hurdles.

I wrote up this wiki page to capture some of the information from here and give us something to link to in the future.

Closing! (unless people still have major gripes of course!)

[m00qek]

Hey everyone, sorry for resurrecting this issue but this weekend I put some thought on this and came up with https://github.com/m00qek/baleia.nvim
Seems to work here in my setup, it is very experimental but you may want to give it a try.

[m00qek]

I think it can be considered stable now https://github.com/m00qek/baleia.nvim#with-conjure

[daveyarwood]

@m00qek I gave those instructions a try and restarted Neovim, but ran into this error:

E5108: Error executing lua /home/dave/.vim/bundle/baleia.nvim/lua/baleia.lua:222: Expected lua string
Error executing vim.schedule lua callback: /home/dave/.vim/bundle/conjure/lua/conjure/buffer.lua:98: Vim(call):E718: Funcref required

[m00qek]

@daveyarwood could you please open an issue there? Strange, baleia.lua does not have 222 lines 😬

[daveyarwood]

I opened that file on my machine, and it has 238 lines!

[daveyarwood]

I wonder if I might have an old version cached, or something.

[daveyarwood]

Aha! That's totally it. I did rm -rf ~/.vim/bundle/baleia.nvim and then reinstalled it and now that file has 83 lines.

[daveyarwood]

It works, both in the HUD and the Conjure log buffer. Awesome work, @m00qek !! 🎉 💯

[daveyarwood]

One thing worth noting is that bold text isn't being displayed as bold. I'll open an issue for that.

[daveyarwood]

Bold is working now too. I'm stoked about this! 💯

[m00qek]

@daveyarwood is it performing well for you? As it needs to watch for changes in the log buffer and then rewrite the changed lines I wonder if it will perform badly in some setups. It would be nice if it was possible to pass a custom vim.api.nvim_buf_set_lines to conjure so it would not be necessary to watch and rewrite lines

[daveyarwood]

I haven't used it much yet, but so far so good! I'll post back here if I notice any performance issues.