Absence of Watermarking

[tftranslate]

The newsrooms have been full of talk about Anthropic and others introducing statistical watermarking into generated text that would even survive copy-pasting and light editing, citing EU law requirements.

From my informed layman's perspective, users of llama.cpp with an OSS model need not worry - statistical watermarking would require initializing the random number generator with specific patterns and there is no evidence that llama.cpp is doing such a thing.

From my professional legal standpoint, neither Hugginface provdiing OSS models nor llama.cpp providing software are required by EU law to do such watermarking as both are out of scope of the EU act. However, If I download a model and run it on llama.cpp and operate a chatbot with these components, then I am in scope, but that's my own problem and not llama.cpp's.

For the informed user, this settles the case. But alas, there is also business reality, there are compliance departments, and someone has asked me to provide "proof" that our local AI does not generate text with hidden watermarks. How do you provide proof of something that is not there?

It would be great if the maintainer could reply to this with a brief competent statement as to whether it is correct that
a) llama.cpp is not and is not currently planning on doing such statistical watermarking, and
b) there is no possibility that a specifically wired OSS model could use existing infrastructure provided by llama.cpp to cause the generation of such statistical watermarking.

[FoxMoss]

Not a maintainer! But I did write an implementation for watermarking in llama.cpp (#27688), as to whether the maintainers want it, no idea. Specifically the Claude style of sampling skewing watermarking is trivial to do, so there's no theoretical blocker for any inference provider to add it.

there is no possibility that a specifically wired OSS model could use existing infrastructure provided by llama.cpp to cause the generation of such statistical watermarking.
You could screw with a models internals to get it to skew outputs that way but that has a greater chance of lobotomizing/worsening a model.

I implemented watermarking since I wanted to verify Anthropic's claims of not worsening outputs, and generally I think disclosure of LLM generated content is good both for the soul and giving people more tools to make sense of content on the internet (a smoking gun you could say.....) So attempting to put my emotional bias aside, I am curious, why you are so worried about llama.cpp implementing watermarking?

[tftranslate]

Thank you for your input. I am generally supportive of the EU AI Act's aim of making sure that users of public and social media should be aware whether they read human- or AI-generated text. If I would operate a public chatbot, I might actually consider pulling your PR. But: I want to know what's going on and want to be in control.

I work in a legal realm where authorship of a text, and authorship of individual ideas within a text, has very high legal relevance. Rights can be granted to new ideas conceived by humans, but not to new ideas conceived by machines under current legal practice. (I am also supportive of that!). However, when I use AI do help me write down an idea that was conceived by a human, and the use of AI aid can be detected by a signature, a malevolent opponent could always try argue the original idea was conceived by the machine and cannot be protected, and countering such a malevolent argument in court is associated with a high price tag. Therefore I want to be sure no such signatures are in my text output.

[FoxMoss]

Hm, I see. I never thought of that!

[tftranslate]

I agree it is a very specific use case, but I am sure there are others that I could never have thought of as well. Therefore, if watermarking goes into llama.cpp, there should be a command line option to turn it off or on. The EU Act only kicks in when you go public as a service provider, not when you use it for your personal purposes.