Project author team stay tuned: I found out that the llama3-V project is stealing a lot of academic work from MiniCPM-Llama3-V 2.5

[pzc163]

Fellow MiniCPM-Llama3-V 2.5 project authors, a few days ago I discovered a shocking fact.There is a large amount of work in the llama3-V (https://github.com/mustafaaljadery/llama3v) project that is suspected to have been stolen from the MiniCPM-Llama3-V 2.5 project, and I raised my query in the GitHub project issue of llama3-v, and did not think that the The authors of Llama3-V quickly deleted my questionable post, and hid Llama3-V's Huggingface project page. I strongly question what they did, and I will release all the evidence next, and I urge you to pay attention to this fact.





this issue has been deleted by the author of llama3-V ( https://github.com/mustafaaljadery/llama3v )，I will expose all the evidence to expose the fact that the authors of llama3-v are a bunch of thieves!

[pzc163]

Fact 1: The llama3-V project uses almost exactly the same model structure and code as the minicom-llama 3-v 2.5 project
Llama3-V has exactly the same model structure and config file as MiniCPM-Llama3-V 2.5, with only the difference in variable names. Left: MiniCPM-Llama3-V 2.5 Right: Llama3-V

Its code appears to be MiniCPM-Llama3-V 2.5's code with some reformatting and variable renaming, including but not limited to image slicing, tokenizer, resampler, and data loading. Just give some examples.

The author of Llama3-V refers to LLaVA-UHD for the architecture, and list difference (on ViT and LLM choice). What the author does not mention is that their specific implementation is identical to MiniCPM-Llama3-V 2.5, which is different from LLaVA-UHD in many ways, such as the spatial schema. Llama3-V also has the same tokenizer as MiniCPM-Llama3-V 2.5, including the special tokens newly defined by MiniCPM-Llama3-V 2.5.

[pzc163]

Fact 2: When I questioned how the authors of llama3-v used MinicPM-Llama3-V2.5's tokenizer before the MinicPM-Llama3-V2.5 project was released, the authors of the llama3-v project began to lie.

The author of llama3-V project thought the tokenizer would be from here: https://huggingface.co/openbmb/MinicPM-V-2/blob/main/tokenizer.json Before llama3 MiniCPM released.
but the fact is that MinicPM-V-2's tokenizer is totally different from MinicPM-Llama3-V2.5，below is the two files in Huggingface. Obviously, they are not the same tokenizer file, and their file sizes are completely different.


And MinicPM-Llama3-v2.5's tokenizer is llama3 tokenizer plus miniCPM-v series model of a few special token composition, and MinicPM-v2 release are before llama3 open source

[pzc163]

Fact 3: The author of llama3-V project afraid to face questioning, deleted the issue I filed at llama3-V questioning their stealing.
Also, it seems the author does not fully understand MiniCPM-Llama3-V 2.5's architecture or their own code. Perceiver resampler is a single-layer cross-attention, not a two-layer self-attention. Sigmoid activation of SigLIP is not used for training multimodal large language models. These activations are only used for pretraining SigLIP.
Llama3-V:

MiniCPM-Llama3-V 2.5:

Visual feature extraction doesn't need sigmoid activation.

[pzc163]

Based on the above three facts, I think there is sufficient evidence to prove that the llama3-v project has stolen the academic achievements of the minicpm-llama 3-v 2.5 project, and I strongly suggest that the minicpm-llama 3-v 2.5 project's team go to the complaint to expose the llama3-v project authors' stealing and lying about academic misconduct, and so on a series of problems!

[Cuiunbo]

Hi @pzc163,
Thank you for sharing this important information with us. We are deeply shocked and will be paying special attention to this matter. We will immediately launch an investigation to verify the above situation. Any new findings will be quickly disclosed to you, to the open-source community, and the public.

This situation sounds extremely serious. We never expected anything like this to happen. We hope the truth will come to light soon.

[pzc163]

Adding two important piece of information：

1. A few days ago, when I tried to run Llama3-V, I found their provided code could not work with their checkpoint from HuggingFace. Many issues about this have been posted on GitHub and HuggingFace, but no reply from the author yet. I changed the variable names in Llama3-V's model weights downloaded from HuggingFace to MiniCPM-Llama3-V 2.5's names, and surprisingly found that the model can be run with MiniCPM-V code successfully.

[model.safetensors.index.json](https://github.com/user-attachments/files/15524692/model.safetensors.index.json)

2.Guess what you get if you add Gaussian noise(parameterized by a single scalar) to MiniCPM-Llama3-V 2.5's checkpoint?

new_dict = {}
for k, v in model.state_dict().items():
torch.cuda.manual_seed_all(42)
new_dict[k] = v + torch.randn_like(v) / 708
model.load_state_dict(new_dict)

That's crazy! You can actually get a new checkpoint, emm, so let's give this new checkpoint a new name and call it llama3-V, doesn't that sound great? At least the hash will be completely different from miniCPM-llama3-V2.5, right?

[yaoyuanTHU]

Thanks for the info. The inference fix and noise sound horrific. We are reproducing it and will test more on some in-house features.

[Cuiunbo]

The conclusion of our investigation:

• Llama3-V can be run using MiniCPM-Llama3-V 2.5's code and config.json after changing param names
• It behaves similarly to MiniCPM-Llama3-V 2.5 in unrevealed experimental features trained on in-house data, e.g., recognizing Tsinghua Bamboo Characters and GUIAgent
• It is somewhat similar to a noised version of MiniCPM-Llama3-V 2.5?

After receiving the issue from @yangzhizheng1on GitHub, we launched a serious investigation. We can obtain inference results correctly using Llama3-V checkpoint with MiniCPM-Llama3-V 2.5's code and config file following @yangzhizheng1's instruction on GitHub. Even more, we also surprisingly find that Llama3-V shows highly similar behaviors to MiniCPM-Llama3-V 2.5 in some unrevealed experimental features, which are trained on private in-house data, such as recognizing Tsinghua Bamboo Characters.

One of the experimental features of MiniCPM-Llama3-V 2.5 is recognizing Tsinghua Bamboo Characters (清华简), a very special and rare type of Chinese ancient characters written on bamboo during China's Warring States Period (475 BC-221 BC). These training images are recently scanned from unearthed cultural relics and annotated by our team, which has not been publicly released yet. Surprisingly, we find highly similar capabilities for Llama3-V in both good and bad cases.

For quantative results, we also tested several Llama3-based VLMs on 1K Bamboo Character images and compared the prediction exact match for each pair of models.

The overlaps between every two models are zero, whereas the overlaps between Llama3-V and MiniCPM-Llama3-V 2.5 achieve a surprising 87%. Moreover, MiniCPM-Llama3-V 2.5 and Llama3-V even share a similar error distribution. Llama3-V and MiniCPM-Llama3-V 2.5 make 236 and 194 wrong predictions respectively, while the overlapped part is 182. The MiniCPM-Llama3-V2.5-noisy obtained following @yangzhizheng1's instruction on GitHub shows nearly identical quantative results with Llama3-V. This is really confusing...

The same thing also happens to WebAgent, another unrevealed feature trained on in-house data. They even make identical errors in a WebAgent schema newly defined within our team...

Since the HuggingFace page of Llama3-V is removed now, we upload the checkpoint here (https://bit.ly/3yRFxYq). Since this model has received several thousands of downloads on HuggingFace, there should be independent copies to reproduce this.

Given these results, we are afraid it is hard to explain such unusual similarities as coincidences. We hope the authors can give an official explanation of the issue. We believe this is important for the common good of the open-source community.

[yu199195]

look～～

[awsaf49]

One of the authors replied to this allegation but deleted the tweet later.

[RylanSchaeffer]

You might want to report this to Stanford CS or Stanford itself. These are serious allegations and they appear (at a quick glance and to my non-expert eyes) to be well substantiated.

[iFe1er]

If the research team from Stanford University is proven to have plagiarized this MiniCPM-V project from Tsinghua University, they should feel ashamed, and also, MiniCPM-V project deserve an apology and acknowledgment.

[Triang-jyed-driung]

You can consult the Dean of Stanford CS department to report misconducts. Refer to this policy:

https://doresearch.stanford.edu/policies/research-policy-handbook/conduct-research/research-misconduct-policy-allegations-investigations-and-reporting

Section 5: Individual Reporting Responsibility
Any individual who believes an act of research misconduct has occurred or is occurring should notify the dean of the appropriate school.

[Triang-jyed-driung]

The current Dean is likely Jennifer Widom:
https://profiles.stanford.edu/jennifer-widom?tab=bio
The one who has the most solid proof should notify her

[motecshine]

https://web.archive.org/web/20240528201635/https://github.com/mustafaaljadery/llama3v/blob/main/model.safetensors.index.json

[larry0x]

Definitely escalate this to Stanford. Plagiarism cannot be tolerated.

[binli123]

If you google Llama3v there have now been more than 1000 pages attributing the work. This has made a detrimental impact. Their actions seemed deliberately planned, aiming for rapid and extensive coverage in tech news with attention-grabbing assertions. This strategy can make the stolen credits be attributed to them quickly before the original authors even realize it. The authors may want to escalate this to their academic administrators immediately to prevent any further negative impact.

[FEIYANG-MAX]

I am not sure if I understand correctly. Actually, the original project is an open-source project, so Llama3-V can use it, but they didn't comply with the open-source license?

[tangmingxing1988]

The latest news, one of the authors Aksh Garg, has acknowledged that on his medium post

We realized that our architecture is very similar to OpenBMB’s “MiniCPM-Llama3-V 2.5...We have taken down our original model in respect to the authors.

[binli123]

The latest news, one of the authors Aksh Garg, has acknowledged that on his medium post

We realized that our architecture is very similar to OpenBMB’s “MiniCPM-Llama3-V 2.5...We have taken down our original model in respect to the authors.

You would hardly be satisfied with this kind of statement if it were your work being copied, with model weights deliberately altered with Gaussian noise and renamed, with a plotted and overwhelming coverage in news and social media (enhanced by eyes-grabbing "$500" statements in the headlines) etc... This goes beyond merely saying "the architecture is very similar to blah blah blah..." And guess what, they said you merely "beat us to the implementation."??? To be honest, one would be furious if they were the authors and saw the statement...

[tangmingxing1988]

A thief is being tried in court, and this is his statement:

"I would like to thank the prosecutor for pressing charges. I realize that my belongings are very similar to those of the victim. To show my respect for him, I am relinquishing these belongings."

[RylanSchaeffer]

https://www.reddit.com/r/stanford/comments/1d75jns/comment/l6x0m4z/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[RylanSchaeffer]

@tangmingxing1988 you forgot the part before the trial where the thief takes a world tour extolling their amazing belongings 😜

[cfpark00]

@pzc163 @Cuiunbo @RylanSchaeffer

Given the two checkpoints, perhaps one can compute the diff in the weight to see the histogram? (even though it already seems like there is enough evidence)

[TomoshibiAkira]

So basically they just randomly added some noise to the weight and called it a day? Sheesh.

[pzc163]

@pzc163 @Cuiunbo @RylanSchaeffer

Given the two checkpoints, perhaps one can compute the diff in the weight to see the histogram? (even though it already seems like there is enough evidence)给定两个检查点，也许可以计算权重的差异来查看直方图？(even虽然已经有足够的证据）
The Llama3-V's HuggingFace page has been removed，you can download its checkpoint from below link:
https://thunlp.oss-cn-qingdao.aliyuncs.com/multi_modal/llama3v.tar
for MiniCPM-Llama 3-V 2.5, here is HuggingFace link: https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5/tree/main

[cfpark00]

Here is a histogram of some random weight's diffs. But I'm no master of fine tuning llms and the resulting weight distribution changes, so I won't draw conclusions myself.....

[RylanSchaeffer]

If you ran a hypothesis test about whether these distributions are Gaussian, what would it tell us?

Agostino-Pearson and Kolmogorov-Smirnov seem reasonable

[RylanSchaeffer]

The fact that all the means are nearly 0 and the standard deviations appear almost identical seems damning...

[pzc163]

Here is a histogram of some random weight's diffs. But I'm no master of fine tuning llms and the resulting weight distribution changes, so I won't draw conclusions myself.....这里是一些随机权重差异的直方图。但我不是微调的大师llms和由此产生的重量分布的变化，所以我不会得出结论自己。

This is the strongest evidence that llama3-V does not train its model at all, but adds random Gaussian noise to the model parameters of miniCPM-llama3-v2.5

[TomoshibiAkira]

Here is a histogram of some random weight's diffs. But I'm no master of fine tuning llms and the resulting weight distribution changes, so I won't draw conclusions myself.....

IMHO no way the delta between a finetuned model with the original would fit a gaussian this well, not to mention that the distribution is almost the SAME across several different layers. Yeah, this is looking very bad.

[TigerHix]

如果只是需要收到新回复提醒可以点击侧边栏的 Subscribe 订阅，无需回复这个 issue，不然所有订阅者都会收到邮件通知。

[RainOfAshes]

Can someone reshare the "weights diff" plot?

[TomoshibiAkira]

https://gist.github.com/TomoshibiAkira/151a2353b946aa9cd8d4d2cdabc31245

Quickly wrote a script to compare the delta of the two weights (LLaMa3V with @Cuiunbo 's link, MiniCPM is from HF), calculate the delta's mean and std and abs(delta)'s mean for every layer, and finally print the histogram across all layers.
You'll need a GPU to run this since some of the them (such as embeddings) are pretty large.

Anyway, here's the final histogram:
Mean of delta:
(array([ 7, 30, 37, 78, 463, 65, 32, 17, 9, 3], dtype=int64), array([-8.14795494e-05, -6.32882118e-05, -4.50968742e-05, -2.69055367e-05, -8.71419907e-06, 9.47713852e-06, 2.76684761e-05, 4.58598137e-05, 6.40511513e-05, 8.22424889e-05, 1.00433826e-04]))
Almost all means of deltas are around 0 across all layers, the maximum is 1e-4.
Since the weight's mean for every layer are around 1e-2 to 1, so the difference is very small.

Std of delta:
(array([ 1, 0, 1, 0, 1, 2, 1, 402, 329, 4], dtype=int64), array([0.00042295, 0.00054622, 0.00066948, 0.00079274, 0.000916 , 0.00103927, 0.00116253, 0.00128579, 0.00140905, 0.00153232, 0.00165558]))
Almost all of them are clustered into 1.2e-3 to 1.4e-3.

Mean of abs(delta):
(array([ 2, 1, 3, 0, 0, 1, 1, 4, 13, 716], dtype=int64), array([6.40153885e-05, 1.76632404e-04, 2.89249420e-04, 4.01866436e-04, 5.14483452e-04, 6.27100468e-04, 7.39717484e-04, 8.52334499e-04, 9.64951515e-04, 1.07756853e-03, 1.19018555e-03]))
Usually the gradients become smaller as the backprop chain gets longer, thus abs(delta) should at least have some "gradual decreasing" type of distribution, but it seems all of them are grouped in 1.07e-3 to 1.19e-3.

[abner63]

你好@pzc163感谢 您与我们分享这一重要信息。我们深感震惊，并将特别关注此事。我们将立即展开调查，核实上述情况。任何新发现都将迅速向您、开源社区和公众披露。

情况听起来非常严重。我们从未想到会发生这样的事情。我们希望真相能尽快大白。

支持维护自身创作权，抄袭令人不齿

[ShiftyBlock]

stanford ai research should issue a statement on this

[RylanSchaeffer]

1. Stanford AI research has zero affiliation with llama3-v

2. Professor Chris Manning already publicly condemned this plagiarism https://twitter.com/chrmanning/status/1797664513367630101

[ShiftyBlock]

Thanks Rylan. From an outsider perspective it seemed so since two authors are/were undergraduate researchers in SAIL. I appreciate you and Professor Manning shedding light on this matter from within the SAIL community.

[RylanSchaeffer]

Were they SAIL researchers?

I'm trying to decide whether to report them to Stanford. I am not personally involved but I think their behavior is clearly immoral and unbecoming of Stanford students.

[ShiftyBlock]

https://www.linkedin.com/in/aksh-garg/ https://www.linkedin.com/in/siddharth-sharma-9942b2104/details/experience/ both point to current/former SAIL experience

[Yuan8341]

Also please report to https://communitystandards.stanford.edu/policies-guidance/honor-code

You can consult the Dean of Stanford CS department to report misconducts. Refer to this policy:

https://doresearch.stanford.edu/policies/research-policy-handbook/conduct-research/research-misconduct-policy-allegations-investigations-and-reporting

Section 5: Individual Reporting Responsibility Any individual who believes an act of research misconduct has occurred or is occurring should notify the dean of the appropriate school.