Repository navigation
Description
Hi, I'm Mark Pilgrim. You may remember me from such classics as "Dive Into Python" and "Universal Character Encoding Detector." I am the original author of chardet. First off, I would like to thank the current maintainers and everyone who has contributed to and improved this project over the years. Truly a Free Software success story.
However, it has been brought to my attention that, in the release 7.0.0, the maintainers claim to have the right to “relicense” the project. They have no such right; doing so is an explicit violation of the LGPL. Licensed code, when modified, must be released under the same LGPL license. Their claim that it is a "complete rewrite" is irrelevant, since they had ample exposure to the originally licensed code (i.e. this is not a "clean room" implementation). Adding a fancy code generator into the mix does not somehow grant them any additional rights.
I respectfully insist that they revert the project to its original license.
Hi Mark, and thank you for reaching out. If you are indeed the Mark Pilgrim, I want to say sincerely that chardet (and Dive Into Python) had an enormous impact on my career and on the Python ecosystem. I've been the primary maintainer and contributor to this project for >12 years and I don't take that stewardship lightly.
I want to address your concerns directly.
The core claim: that this is not a clean-room implementation and therefore must remain LGPL.
I understand the concern, and I think it's worth engaging with seriously. You're right that I have had extensive exposure to the original codebase: I've been maintaining it for over a decade. A traditional clean-room approach involves a s…
Activity
schuelermine commentedon Mar 4, 2026
Is there a fork from before the AI-assisted “rewrite” happened?
paulRbr commentedon Mar 5, 2026
All versions before v7.0.0 are visible by tags in this repo. E.g. here's version 6.0.0
wmealing commentedon Mar 5, 2026
You dont get the work horse, without having to clean the stable. Its the tradeoffs we make.
jaens commentedon Mar 5, 2026
Google LLC v. Oracle America, Inc.
"...the Federal Circuit court reversed both decisions, holding that APIs are copyrightable in 2014..."
(That makes the relicensing an API-compatible "rewrite" without the copyright holder's permission likely illegal in the US unless it's fair use, which is how Google was "saved". This rewrite is very unlikely to meet all the requirements for fair use.)
flexagoon commentedon Mar 5, 2026
@jaens IMO this has less to do with APIs and more with what a2mark originally mentioned - the rewrite is still heavily based on the original code, and as such is still a derivative work
mitsuhiko commentedon Mar 5, 2026
That is neither what the ruling said nor what would matter here. Google was very much within their rights the match a public API and the same would most likely be true for an API compatible rewrite of chardet.
If the current rewrite can be seen as a clean room implementation I cannot say. Chardet was already on the list of libraries I think should have an attempted clean room reimplementation, because of both performance and licensing reasons.
It’s very much within what should be possible by having an agent implement by just targeting inputs and outputs without setting the source.
So if this does not yet match that, it should be possible, and I think it should at least be seriously attempted.
justinclift commentedon Mar 5, 2026
As reference info, this is the MIT license in this repo (ie the new license): https://github.com/chardet/chardet/blob/dc43ec8f95b74458effd38f9c076a540c6990049/LICENSE
theronic commentedon Mar 5, 2026
@jaens,
justinclift commentedon Mar 5, 2026
On that note, should it be assumed (or confirmed?) that the chardet source code was part of the training data for the LLM(s) involved in the rewrite?
If it was, that could be extremely relevant.
therealadityashankar commentedon Mar 5, 2026
I have to admit this sounds like an argument to why licensing is useless altogether due to an agent always being able to reconstruct a project on the basis it's inputs and outputs, which may be true, but I think this opens the pathway to many more questions about the point of licensing in the first place
jaens commentedon Mar 5, 2026
@theronic I literally mentioned that fair use saved Google.
The Supreme Court did not take a stance on API copyrightability so it's sane to conservatively assume that the example set by the Federal Court holds until proven otherwise.
If anybody thinks this is fair use, they could explain exactly how this meets the four factors considered in fair use.
mitsuhiko commentedon Mar 5, 2026
I think the interesting question here right now is if we can use this under MIT safely and the question about the greater implications on copyright or licensing should be have elsewhere. I personally welcome an MIT (or public domain) implementation of chardet so I have some vested interests in this new library. So far at least it looks to me that this is a clean room implementation that neither shares algorithm nor code with the existing implementation.
Right now I would argue that unless some evidence of the contrary could be provided, this can be seen as a new implementation from ground up.
ii14 commentedon Mar 5, 2026
I have not investigated this any further, but the agent was looking at some of the original files, according to its plan: 925bccb#diff-704d0561e3e878b036836b56870212b2d8b7e4a813bf2805df868a1bf9e0094cR309-R314
mitsuhiko commentedon Mar 5, 2026
@ii14 those are mappings: https://raw.githubusercontent.com/chardet/chardet/f0676c0d6a4263827924b78a62957547fca40052/chardet/metadata/charsets.py
225 remaining items
ell1e commentedon Mar 7, 2026
I still don't get why lawyers don't also raise a red flag on any LLM use whatsoever, even outside of rewrites like this, given it's (apparently?) been shown to accidentally and fully unprompted plagiarize large individual things: https://www.pcgamer.com/software/ai/microsoft-uses-plagiarized-ai-slop-flowchart-to-explain-how-github-works-removes-it-after-original-creator-calls-it-out-careless-blatantly-amateuristic-and-lacking-any-ambition-to-put-it-gently/
Pannoniae commentedon Mar 7, 2026
I'm not surprised that an auditor would say this, considering their incentives are aligned on having issues to find :P
Nothing, welcome to the post-copyright, post-execution barrier world, where anything can be recreated with a bunch of API credits. Open software licences were created on the presumption that reimplementing stuff (and especially doing it clean-room!) is WAY more expensive than compliance.
The GPL wouldn't exist today if Stallman could just vibecode that printer driver. :)
This kind of flips it on the head though, if it's trivial to reimplement then people will just be doing that, doesn't matter if it's GPL or proprietary. Why would anyone pay for software when they can reimplement it? (even if poorly)
maslade commentedon Mar 7, 2026
My guess is that the ones who do get bowled over by the apparently unstoppable momentum of AI.
ell1e commentedon Mar 7, 2026
Is it unstoppable, or is that just what some hype investors want you to think?
dan-blanchard commentedon Mar 7, 2026
At this point, I think all of the original thoughts that were going to be shared on this issue have been shared, so I'm going to lock this. If people want to have more debates, there are other mediums that are better suited to this
dan-blanchard commentedon Mar 26, 2026
I'm going to agree with this comment from one the author's of LGPLv3 that this project does not legally need to be licensed under LGPL and close this: #334 (comment)