Rich markdown formatting (including streaming) in any mode with --rich

[gianlucatruda]

Overview

Fixes #12

This builds on the excellent foundation that @juftin laid down in #278 and

1. fixes a mysterious bug that was failing some tests and
2. resolves merge conflicts caused by changes since rich printing #278 was proposed.

I love llm and use it constantly. My only gripe has been the lack of rich formatting in the terminal. I recently used rich for a project and found it excellent, so I was excited to add this to llm. I found #278 was open but dormant, so I decided to nudge things along.

@simonw thanks for your amazing tools and awsome blog!

Screenshots

[gianlucatruda]

Here's a demo gif of streaming working with rich output:

[gianlucatruda]

Update: I've added pytest tests for making sure that --rich mode works as intended. I also factored in release 0.16 commits. All 185 tests pass.

@simonw is there anything else this needs in order to be merged? That would allow you to close #12

[irthomasthomas]

What is the benefit versus piping to something? I like the idea of keeping the main project as light as possible.

[dzmitry-kankalovich]

@irthomasthomas likely the difference is in syntax highlighting of a partial / streamed LLM response.

I've been piping llms output to glow for the past months, but the drawback is that you see the result only when streaming is completed, and until that there is like no output. It's a subpar UX when you need to wait some dozens of seconds to see the result.

As I get it from @gianlucatruda examples here this particular problem was solved.

[gianlucatruda]

I like the idea of keeping the main project as light as possible.

I normally would agree, @irthomasthomas. But as @dzmitry-kankalovich correctly points out, piping breaks streaming, which is a major drawback to usability. I think this justifies the choice.

[irthomasthomas]

Are you talking about the ansi codes being injected?

…

On Mon, 16 Sept 2024, 18:49 Gianluca Truda, ***@***.***> wrote: I like the idea of keeping the main project as light as possible. I normally would agree, @irthomasthomas <https://github.com/irthomasthomas>. But as @dzmitry-kankalovich <https://github.com/dzmitry-kankalovich> correctly points out, that breaks streaming, which is a major drawback to usability. I think this justifies the choice. — Reply to this email directly, view it on GitHub <#571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE476NE4CDZSB6EHNVAT3ULZW4K37AVCNFSM6AAAAABOD5UQFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJTGU2DINZXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gianlucatruda]

Are you talking about the ansi codes being injected?

@irthomasthomas When you pipe the output of llm to another application that renders markdown (which may do ANSI code injection), you have to wait for the entire LLM response, which could be several seconds or even minutes. And in chat mode, it's not possible at all. So it's not a viable solution.

This PR enables llm to do the rich textual rendering itself in a way that supports response streaming. That means the user sees the llm output in realtime, rendered prettily, as it arrives from the LLM. It also allows this rich text streaming to work in chat mode (as seen in my screenshots).

Overall, this PR adds functionality to llm that is not possible when piping to other tools. It's a massive upgrade to the user experience and something that has been requested by many people for a long time.

[irthomasthomas]

That's not true. The example I gave, I'm using highlight and that displays the rendered markdown as it streams in.

…

On Mon, 16 Sept 2024, 20:16 Gianluca Truda, ***@***.***> wrote: Are you talking about the ansi codes being injected? @irthomasthomas <https://github.com/irthomasthomas> When you pipe the output of llm to another application that renders markdown (which may do ANSI code injection), you have to wait for the entire LLM response, which could be several seconds or even minutes. And in chat mode, it's not possible at all. So it's not a viable solution. This PR enables llm to do the rich textual rendering itself in a way that supports response streaming <#571 (comment)>. That means the user sees the llm output in realtime, rendered prettily, as it arrives from the LLM. It also allows this rich text streaming to work in chat mode <https://private-user-images.githubusercontent.com/1952799/367031734-b556db5f-838c-4544-9091-429a8e199b3c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjY1MTM4NTEsIm5iZiI6MTcyNjUxMzU1MSwicGF0aCI6Ii8xOTUyNzk5LzM2NzAzMTczNC1iNTU2ZGI1Zi04MzhjLTQ1NDQtOTA5MS00MjlhOGUxOTliM2MucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDkxNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA5MTZUMTkwNTUxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NGViNzY4MTZmZmQ2NmQxODAxNTAxMmNjZTMyYmI4OWJkNWNlNmZiZWRmYjUzZjJmMGUyNmM2MThmNzQ0ZmRhMSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.N-bsTtyPFqoavRBubZW8xg8_I2X3p7u5c4xy5A4Zxfw> . Overall, this PR adds functionality to llm that is not possible when piping to other tools. It's a massive upgrade to the user experience and something that has been requested by many people for a long time. — Reply to this email directly, view it on GitHub <#571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE476NDTOGGRZ4EPXPSHXT3ZW4VAVAVCNFSM6AAAAABOD5UQFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJTG4ZDANJZGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dzmitry-kankalovich]

@irthomasthomas I just checked, and indeed highlight unlike glow does process stream responses.

however it... does not render markdown?

it just highlights (I guess hence the name) markdown blocks, but it does not render them - at least not like glow or rich.

It is somewhat better than just plain text, but it does not feel as convenient as these other alternatives.

[gianlucatruda]

That's not true. The example I gave, I'm using highlight and that displays the rendered markdown as it streams in.

@irthomasthomas

1. can you link to the source for installing highlight? If it's cli-highlight, then I'm unable to replicate the streaming you claim.
2. your screenshots with piping to highlight are showing syntax highlighting, not markdown rendering as Markdown renderer support  #12 is asking for and this PR provides.
3. can you provide an example showing evidence that streaming works with piping in both normal and chat modes?

[gianlucatruda]

@simonw let me know if you have any feedback on this PR. Happy to make any changes necessary.

[sa-]

This could also work

live.update(
    Markdown(full_response)
    if rich
    else Text(full_response)
)

[gianlucatruda]

UPDATE: Since Simon doesn't seem to want to review this, a few of us in #12 have been trading workarounds. See here where I talk about richify. It's the same core functionality, but with some tweaks to the styling of the output and uv's script runner mode to automatically install and isolate dependencies -- making it much nicer and simpler to run as a standalone script.

Feedback and contributions are actively welcomed!

[irthomasthomas]

I use highlight

llm "prompt" | highlight --syntax=bash --out-format=ansi

[bretello]

This would be a pretty nice improvement to have

[vphantom]

Since this is an opt-in feature, why is it not being considered for merging?

Coming from Aider, I really like streaming Markdown formatting, including code blocks, and I was hoping to be able to replicate the experience with LLM's interactive mode. The piping workarounds mentioned here only help in one-shot mode.

[gianlucatruda]

Since this is an opt-in feature, why is it not being considered for merging?

I agree, @vphantom, but @simonw has never responded to this PR or previous PRs/issues relating to this functionality. I tried contacting him on other platforms over many months as well, to no avail. Given that the contributions guide is there (and we followed it), it would be nice to get some kind of response, but this is open source and he's under no obligation to anyone.

The piping workarounds mentioned here only help in one-shot mode.

I use richify.py as a stop-gap solution (though it doesn't work in chat mode, as you mention). I still use it constantly and it annoys me every time that this isn't just a flag option in llm 😢

I've thought about just forking or re-writing llm with rich rendering functionality, but then I'd have to constantly update it to keep up with evolving models and APIs and I don't love the prospect of taking that on.

Someone once suggested trying to add the rich rendering functionality via plugins, but I suspect that to be an ass-pain.

[gianlucatruda]

It looks like there's a recent open PR proposing an almost identical solution, with very similar code: #981

[gianlucatruda]

I also found a fairly recent comment from Simon relating to this:

I don't want to bake this stuff into LLM core but I'd love to enable plugins to do custom stuff like this.

Hoping to get some clarity there

[ldelossa]

Ouch, not providing a 'streaming' formatter, when the main application defaults to streaming text, is a pretty bad take from the authors

[gianlucatruda]

Life is too short