criminals could use this to violate the EU AI Act

[Lewiscowles1986]

This is really interesting; but also probably short sighted. I found out about the hidden UTF8 in chat transcripts yesterday and today Facebook links this repo.

Sovereignty aside for individuals; there will be bad actors who are acting as criminal enterprise, who will use this to harm people.

Is it smart to bundle as an "all in one" (AIO) utility knowing this? It would make a great set of whitepapers or blog articles; but this is dangerous assembling the pieces for people surely?

[transfairs]

What use cases do you see for this to be used in a harmful context?

The new normal is that companies will leverage AI for their legitimate content which then will be flagged as AI generated. Scammers will find excuses, e.g., claiming that this is a human, but the text is translated via AI. They will say that AI was included in creating this content, but a human in the loop was involved.

"Normal" people won't be able to use an AI detector and won't know whether this content is AI generated. Scammers will build their own tools to hide their traces.

This tool is not a weapon, similar to a car (or a VPN) that could also be used in a harmful way which is not a reason to ban cars/VPNs. This tool leverages freedom.

Benjamin Franklin:

Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.

[Lewiscowles1986]

Ben Franklin also said: [Ben did not say this, search lied, I never checked]
> The industrious man needs no food, for there shall be nourishment enough in the grave

I'm guessing you still eat?

My whole point is that I dont think criminals are smart or dedicated enough to build something like this and most are not intelligent enough to glue it all together if it existed separately. Nation state actors occasionally get people to work towards negative ends.

In any case this is less like a car; it's not like stripping DRM (which I'm all for); and I am very interested in the author thinking about potential harms from this work.

[ME0Mahmoudi]

Criminals are more dedicated than normal people and often times, smarter. What makes you think criminals aren't intelligent?

Also, what harm do you think criminals will use this for? That question remains unanswered

[guillaumemeyer]

This is an important question @Lewiscowles1986 .

First, my intent: My only goal with this project was educational.

Now, in practice, technically, "could use this to violate the EU AI Act", but it would not respect what's clearly stated in the first line of the README, aka for "...privacy and hygiene on content you own".

I think the best answer is already in our ethics document.
It may need some updates, and I am very open to suggestions.

[Lewiscowles1986]

thanks @guillaumemeyer I'll close out, I just wanted to raise it as a concern; but it is very educational.

[guillaumemeyer]

Switching to a discussion instead of an issue.

[BradyDouthit]

I'm fully aware of all of the malware etc. on Github, so by my logic this is no different. I also know anyone can put whatever they want on here. That said, what possible good could come from hiding indicators that text is AI generated? Please enlighten me to what this is for, because it does seem like its main usefulness would be in misleading people, spreading misinformation etc.

Sometimes tools built with good intentions are still used for bad, but the author is still the one putting it out into the world. Merely stating it should be used for "education" is just covering yourself.

[transfairs]

Sure.

1. Allegations: You created a thousand pages document and AI wrote 10 pages of it? With a watermark, there is no difference if it was the other way around. You wrote the perfect app and let AI deliver the translations versus AI wrote the whole app. No plausible way to distinguish.
2. Adoption obstacle: People creating great products don't want that watermark in their product. So, they refuse to ask AI for a security/performance/usability review.
3. You don't have a choice: The watermark is in your files, no matter what, and AI won't tell you about it. This is the first step where AI is used against you. What comes next? Report about the files that a user has stored in his computer? Different story if you would get asked each time and you have a real choice to say no.
4. Overhead: AI could do a better job. Putting this watermark in deviates the output from the ideal path. The output is worse than without it, execution is slower, and token consumption most likely higher. It needs to get maintained and the overhead consumes real world resources.
5. Education is valid: You need to know what is possible. If a scammer tricks you with an AI tool without a watermark, you may fall for it as you are not aware that the watermark can be removed. Someone claims that this product is made by himself. Without proper checking as you trust the missing watermark, cheating in exams will be easier.

A watermark is not a solution for the given problem: It is hidden, it can be removed, it will either be identified when it is too late already (scam) or even never (clickbait that no one cares about).

[guillaumemeyer]

💯

[Lewiscowles1986]

I think the points are valid, even if the research (which is still interesting) is only intended for that. To some extent I wonder how much intentions matter.

[rambip]

I'm really glad this discussion exists. I strongly disagree with the anti-watermark arguments.

I want to respond as thoughtfully and open-mindedly as I can to the argument, and then present my own.

Allegations: You created a thousand pages document and AI wrote 10 pages of it? With a watermark, there is no difference if it was the other way around. You wrote the perfect app and let AI deliver the translations versus AI wrote the whole app. No plausible way to distinguish.

It's not technically true. You can measure the amount of AI-watermark signal in the text quite precisely, in fact. For a given domain (say a report), the strength of the watermark in bits will be roughly proportional to the number of words added or edited by the AI. If your problem is translation, then do not use a LLM for translating! Translation models exist and do not require a watermark.
I'm pretty sure the company will have an API to measure the strength of their signal (modulo cryptographic key if the watermark uses one; not sure about that). So you could say something like "10% of the paragraphs of this text look AI-generated)

Adoption obstacle: People creating great products don't want that watermark in their product. So, they refuse to ask AI for a security/performance/usability review.

Do you have a more precise answer in mind? It's actually what happens for most industries: if you ask some company to build this piece or this service for you, it's very likely that the end user will be able to guess it, with enough motivation.
And the "review" argument does not make sense to me: the security review will never be included in the final app, so why is it a problem that there is a watermark? Nobody will ever see it, because you don't "ship" a review ...

You don't have a choice: The watermark is in your files, no matter what, and AI won't tell you about it. This is the first step where AI is used against you. What comes next? Report about the files that a user has stored in his computer? Different story if you would get asked each time and you have a real choice to say no.

These are not "your" files; these are, by definition, the files generated by the LLM. The LLM can't even add the watermark to existing files, this is very often not possible without changing the content. If you provide a specific example of how it could be used against the user, maybe I would see your point, but for now I don't see it.
And when you're using an AI agent, every image and text the model sees is actually stored by the LLM provider. This is not even something the providers are trying to hide; it's written in their terms of use.

Overhead: AI could do a better job. Putting this watermark in deviates the output from the ideal path. The output is worse than without it, execution is slower, and token consumption most likely higher. It needs to get maintained and the overhead consumes real world resources.

Turns out it's not. Papers have measured it,in particular the google synthID one

Quality of text (measured by human judges) decreased by 0.01% and 0.5% in speed, which is absolutely nothing. LLM providers often use a temperature of around 1, and with such a high temperature, there is a lot of room to put your watermark without degrading quality.

Education is valid: You need to know what is possible. If a scammer tricks you with an AI tool without a watermark, you may fall for it as you are not aware that the watermark can be removed. Someone claims that this product is made by himself. Without proper checking as you trust the missing watermark, cheating in exams will be easier.

It's the only point I slightly agree on. It can actually create incentives for the providers to create more robust watermarks, that you will not be able to remove. But 1. I don't think it outweighs the risks and 2. you should make clear that it's your goal, first thing in the readme. with a big disclaimer it's currently not the case.

Pro-watermark arguments

This profits to criminals

This kind of open-source project is used at least as much by criminals than by honnest people. It has always be the case, I'm thinking of:

• https://github.com/elder-plinius/OBLITERATUS, to remove the safety guardrails of AI agents
• https://github.com/hacksider/Deep-Live-Cam clearly used for AI fraud and DeepNudes (this has been documented here: https://www.canva.com/design/DAGLHpt6WlY/Hfztqtw_-tKza_2l1cPNrA/view?utm_content=DAGLHpt6WlY&utm_campaign=designshare&utm_medium=link&utm_source=editor#3)

In another domain, https://github.com/harry0703/MoneyPrinterTurbo which is a AI-slop generator.

Watermarks are our last hope to save human creators

On youtube, major human-centered youtube channels have lost a lot of their views because some of their videos have been categorized as AI-generated. That's the case for Kurzgesagt and RealEngineering. Why? Because some AI-slop channels have tried to imitate their style and their voice, leading to youtube AI detection algorithm being confused. The original creators are desperate for a way to prove that the plagiarists are using AI to make slop from their work. Watermarks may be the only way to improve the situation. If projects like your gain traction, and if everyone starts to think "watermarks are bad, we should remove them", then plagiarists will use those open-source project to remove the watermarks, and all this effort will be useless.

I am aware that if you did not created this repo, someone else would have done it. But at least, you should put a clearer disclaimer IMO, because watermarks are really solving a big problem.

[Lewiscowles1986]

On youtube, major human-centered youtube channels have lost a lot of their views because some of their videos have been categorized as AI-generated. That's the case for Kurzgesagt and RealEngineering.

I dont know if the plight of YouTube content creators is the hill I wish to die on... In fact I think they made slop before AI ever existed

[rambip]

But at least plagiarists could not press a button, have a complete article written for them in the style of someone else, and get some percentage of the recognition of the original hard-workers.

[Lewiscowles1986]

For me it's more about law than morality; OpenSource can be immoral for people contributing who are unpaid. Pay and recompense are not coupled to value or outcomes

[Lewiscowles1986]

People creating great products don't want that watermark in their product.

If they want to sell in the EU, they legally have to not just watermark, but make it clear that their product is AI

[guillaumemeyer]

I wrote an article to summarize my stance on this issue: https://x.com/guillaumemeyer/status/2088779579733524634
Hope it contributes to the debate.

[Lewiscowles1986]

We are aligned on that sentiment. I didn't see them as the same. So is part of this also that there is no attribution to your guidance and involvement, or is it that you use LLM so little, that it's the case you still write most of the code and LLM does less than 20%?

[guillaumemeyer]

1. I work my ass off to produce interesting content and tools, whether AI-generated or not. I use AI for 99% of what I do online; disclosing that isn't an issue, and I'm all for content attribution.
2. Also, I am not a native English speaker, so I use Grammarly intensively for proofreading; does that mean 100% of what I do should be watermarked as AI-generated? I don't think so.
3. Finally, and maybe more importantly, as someone interested in both history and philosophy, the idea of organizations labeling or marking ideas feels very wrong.

[SyedUmerHannan]

Every law ever passed has loopholes and bypass strategies, such is the world we live in.

[guillaumemeyer]

Cat and mouse game. Even if, for this project specifically, it's only for educational purposes, of course...

[ME0Mahmoudi]

So did all those against this fail to provide examples of how this can be harmful? I'm sorry but is it the way now to judge even before anything happens and based on assumptions?
I didn't see a reason provided on why having no watermark will be harmful. Also, if it is harmful, then force other AI providers to put watermarks to "prevent harm". You see how that doesn't happen? Because people aren't harmed by not having a watermark in AI, and also, there'll always be harm. Doesn't mean we have to restrict everything.
Original youtubers being considered AI users is a mistake on those claiming these allegations wrongly. Doesn't mean normal people did anything wrong to be confined. They should have a choice. Do teachers force students to put watermarks in their knowledge and tell it to everyone whenever they use a bit of knowledge they got from others? No.
People talk about original content. What is that? We all use recycled knowledge anyway

[Lewiscowles1986]

I didn't see a reason provided on why having no watermark will be harmful.

Nobody but you has disputed the potential harms. An Entire Act has passed in the largest single economic block. It's lazy.

there'll always be harm. Doesn't mean we have to restrict everything.

Nobody said we restrict everything, but we do restrict, trains, planes, automobiles, bullets, bombs,and to some extent electronics hardware itself. This is because in terms of leverage, they all allow outsized impacts. Totally daft that because we dont need to restrict everything you are arguing to restrict nothing...

Do teachers force students to put watermarks in their knowledge and tell it to everyone whenever they use a bit of knowledge they got from others? No.

You literally just described higher education citations; and then said it does not happen

People talk about original content. What is that? We all use recycled knowledge anyway

We agree on this, and potentially on the wasted time, effort and thought put into "who's idea was it anyway?"; but we live in the world we do live in, not the one we want to live in.

Most of the comments I've seen are nonsense. The most valuable one is from the author, who said plainly. I wanted to experiment; and I'm grateful for their sharing; and I accept that they have decided that their curiosity should not be limited. Hopefully this thread helps more responsible thoughtful people to also think before they use this in a commercial or criminal setting.

[rambip]

Examples of harm that could be caused by this project:

• Can be integrated into a sophisticated phishing pipeline. Phishing is increasingly hard to detect, and watermarks could help a lot with that. Removing watermarks would break this solution.
• Could be integrated into a tool to create AI-slop websites to avoid detection (say, by Google search)
• Could be integrated into a tool to create AI-slop LinkedIn posts to avoid detection (say, by LinkedIn)
• Could be integrated into a tool to create AI-slop X posts to avoid detection (say, by X)
• Could be integrated into a tool to create AI-slop papers to avoid detection (say, by arxiv)
• Could be integrated into a tool to create any AI content to avoid detection (by anything)
• Integrated into a nice tool that students can use to remove evidence that their homework was AI-generated (agreed, it's somewhat low probability. But at least I hope we agree that it's good for teachers to know what proportion of the text was AI-generated). Teachers then have no way to prove that the text is AI-generated. If teachers discover it is possible to break the watermark, then the trust between teachers and students that the watermark was supposed to create is impaired.
• Can be used by anti-AI-regulation lobbyists as proof that people don't want a watermark (which is largely false).

Note that one reason why it's hard to come up with convincing harms is that watermarks is just being integrated into the industry, so it does not break anything, FOR NOW. I agree with @Lewiscowles1986 that it's unfair to ask more evidence for the PRO-side than the ANTI-side.
Note also that I agree with the argument of @guillaumemeyer

the idea of organizations labeling or marking ideas feels very wrong.

But in this context, it's not labelling people's ideas, and I think this makes a big difference. If you use chatGPT for thinking through your idea, then it's still your own, and no problem with that. If you want to share your idea, then simply type it on your keyboard, dictate it, translate it if you're not fluent in the language ... All of these options won't add a watermark.