Adding linenoise.cpp to llama-run by ericcurtin · Pull Request #11252 · ggml-org/llama.cpp

ericcurtin · 2025-01-15T13:29:25Z

This is a fork of linenoise that is C++17 compatible. I intend on adding it to llama-run so we can do things like traverse prompt history via the up and down arrows:

https://github.com/ericcurtin/linenoise.cpp

ericcurtin · 2025-01-15T13:30:23Z

Linked issue:

containers/ramalama#586

rhatdan · 2025-01-15T19:54:31Z

This does not work on Windows? I guess inside of a container it would work.

ericcurtin · 2025-01-15T20:01:07Z

This does not work on Windows? I guess inside of a container it would work.

I couldn't find any library like linenoise that had Windows support. Plenty have macOS/Linux support. But yeah if using a container or WSL2, etc. we'd be fine.

I didn't kill Windows native support here, but you don't get the cool features like being able to cycle through prompt history with up and down arrows.

slaren · 2025-01-15T20:13:38Z

Note that this will require adding a notice in the releases.

Where can I copy that in? I think the license is compatible once we include these copyrights

Not sure how this is usually handled, I guess we could add a file with all 3rd party copyright notices and copy it to the release packages.

We have the same issue with CURL if we start distributing it for Windows FWIW, from curl license: "Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies."

@ggerganov you were speaking about shipping curl on Windows recently.

I also don't know what is the common way to handle this. Probably look for how other projects do it.

Btw, each llama.cpp release already includes a copy of the full source code in this repo so I think all licenses are technically already included in the release packages? Again, we can make this more explicit if necessary - feel free to improve.

I'm ok with that technique, there's no standard technique. Everybody does it a little different.

I am not a lawyer, but I am not convinced that's enough. We already copy our own LICENSE file to the release packages as llama.cpp.txt, so it shouldn't be a problem to also add the 3rd party licenses there.

@slaren can you show me the script where that copy occurs? We can just check in:

https://github.com/ericcurtin/linenoise.cpp/blob/main/LICENSE

and copy that file to linenoise.cpp.txt maybe ?

https://github.com/ggerganov/llama.cpp/blob/master/.github/workflows/build.yml

Thanks my searching tool skips hidden directories, sometimes I forget this. I hope it's good now, it should do the copy of the LICENSE now I think.

rhatdan · 2025-01-15T21:14:28Z

Most of the time windows will be using a container so this should not be an issue.

ericcurtin · 2025-01-16T12:25:52Z

What unblocks this from getting merged @slaren and @ggerganov ?

ericcurtin · 2025-01-16T20:15:38Z

Yay green build and licensing file copied @slaren @ggerganov

slaren · 2025-01-16T23:46:16Z

This will only affect the Windows releases, but the macOS and linux releases should also be updated.

How about now?

ericcurtin · 2025-01-17T12:47:59Z

Haven't been able to reproduce the other one.

ngxson · 2025-01-17T13:08:59Z

The error with model load could be a false positive described in https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow , where one part of the app is built without ASAN enabled (could be gguf.cpp in this case)

@ggerganov Did you try deleting the build folder?

ericcurtin · 2025-01-17T13:11:14Z

Nonetheless I pushed the list change 😄

This is a fork of linenoise that is C++17 compatible. I intend on adding it to llama-run so we can do things like traverse prompt history via the up and down arrows: https://github.com/ericcurtin/linenoise.cpp Signed-off-by: Eric Curtin <ecurtin@redhat.com>

slaren · 2025-01-17T13:14:06Z

The error with model load could be a false positive described in https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow , where one part of the app is built without ASAN enabled (could be gguf.cpp in this case)

Now that you mention it, the llama.cpp files are built without sanitizer flags. This needs to be fixed as well.

ericcurtin · 2025-01-17T13:15:07Z

The error with model load could be a false positive described in https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow , where one part of the app is built without ASAN enabled (could be gguf.cpp in this case)

@ggerganov Did you try deleting the build folder?

You could be right, things do run fine without asan

ggerganov · 2025-01-17T13:15:15Z

The error with model load could be a false positive described in Wiki: AddressSanitizerContainerOverflow (google/sanitizers) , where one part of the app is built without ASAN enabled (could be gguf.cpp in this case)

Now that you mention it, the llama.cpp files are built without sanitizer flags. This needs to be fixed as well.

I'll push a fix on master.

ngxson · 2025-01-17T13:16:31Z

For the msg_strs problem, I would prefer constructing the vector/list inside apply_chat_template because it's more aligned with functional programming approach. common_chat_apply_template does the same thing too (though, it's up to you to decide if you want to use or not)

ggerganov · 2025-01-17T13:24:24Z

#11279 resolves the sanitizer error on mac.

JohannesGaessler · 2025-01-18T08:03:06Z

Is my understanding correct that the issue with the GGUF code turned out to be a false positive that has now been fixed?

ericcurtin · 2025-01-18T10:19:12Z

Is my understanding correct that the issue with the GGUF code turned out to be a false positive that has now been fixed?

Yes and I think all the issues discussed in this PR are resolved now.

This is a fork of linenoise that is C++17 compatible. I intend on adding it to llama-run so we can do things like traverse prompt history via the up and down arrows: https://github.com/ericcurtin/linenoise.cpp Signed-off-by: Eric Curtin <ecurtin@redhat.com>

github-actions Bot added the examples label Jan 15, 2025

ericcurtin force-pushed the linenoise.cpp branch 8 times, most recently from fdaa9ba to bd3e9ae Compare January 15, 2025 17:23

arch-btw mentioned this pull request Jan 15, 2025

Feature request: Up/down arrow to cycle through previous messages #11182

Closed

ericcurtin force-pushed the linenoise.cpp branch 2 times, most recently from 14dc5f3 to 29d40de Compare January 15, 2025 19:49

ericcurtin force-pushed the linenoise.cpp branch from 29d40de to 47487ae Compare January 15, 2025 20:12

slaren reviewed Jan 15, 2025

View reviewed changes

ericcurtin force-pushed the linenoise.cpp branch 2 times, most recently from 27812c8 to 6360829 Compare January 15, 2025 20:52

ericcurtin force-pushed the linenoise.cpp branch 3 times, most recently from 4127d6a to 1aa42d3 Compare January 16, 2025 10:38

ericcurtin force-pushed the linenoise.cpp branch from 1aa42d3 to e3f66a8 Compare January 16, 2025 17:33

github-actions Bot added the devops improvements to build systems and github actions label Jan 16, 2025

ericcurtin force-pushed the linenoise.cpp branch from e3f66a8 to e2d118a Compare January 16, 2025 19:22

slaren reviewed Jan 16, 2025

View reviewed changes

ericcurtin force-pushed the linenoise.cpp branch from e2d118a to 05c338b Compare January 16, 2025 23:54

ericcurtin force-pushed the linenoise.cpp branch from 33d1eec to 164c4ee Compare January 17, 2025 13:10

github-actions Bot added build Compilation issues ggml changes relating to the ggml tensor library for machine learning labels Jan 17, 2025

ericcurtin force-pushed the linenoise.cpp branch from 164c4ee to 0ea354c Compare January 17, 2025 13:13

ggerganov mentioned this pull request Jan 17, 2025

cmake : add sanitizer flags for llama.cpp #11279

Merged

slaren approved these changes Jan 18, 2025

View reviewed changes

ericcurtin merged commit a1649cc into ggml-org:master Jan 18, 2025

ericcurtin deleted the linenoise.cpp branch January 18, 2025 14:42

Conversation

ericcurtin commented Jan 15, 2025

Uh oh!

ericcurtin commented Jan 15, 2025

Uh oh!

rhatdan commented Jan 15, 2025

Uh oh!

ericcurtin commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericcurtin Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericcurtin Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhatdan commented Jan 15, 2025

Uh oh!

ericcurtin commented Jan 16, 2025

Uh oh!

ericcurtin commented Jan 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericcurtin commented Jan 17, 2025

Uh oh!

ngxson commented Jan 17, 2025

Uh oh!

ericcurtin commented Jan 17, 2025

Uh oh!

slaren commented Jan 17, 2025

Uh oh!

ericcurtin commented Jan 17, 2025

Uh oh!

ggerganov commented Jan 17, 2025

Uh oh!

ngxson commented Jan 17, 2025

Uh oh!

ggerganov commented Jan 17, 2025

Uh oh!

JohannesGaessler commented Jan 18, 2025

Uh oh!

ericcurtin commented Jan 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ericcurtin commented Jan 15, 2025 •

edited

Loading

ericcurtin Jan 15, 2025 •

edited

Loading

ericcurtin Jan 15, 2025 •

edited

Loading