Skip to content

Commit

Permalink
Tests: Add a few test files.
Browse files Browse the repository at this point in the history
  • Loading branch information
JiaT75 committed Feb 23, 2024
1 parent 39f4a1a commit cf44e4b
Show file tree
Hide file tree
Showing 6 changed files with 19 additions and 0 deletions.
19 changes: 19 additions & 0 deletions tests/files/README
Expand Up @@ -41,6 +41,8 @@
good-0catpad-empty.xz has two zero-Block Streams concatenated with
four-byte Stream Padding between the Streams.

good-2cat.xz has two Streams with one Block each.

good-1-check-none.xz has one Stream with one Block with two
uncompressed LZMA2 chunks and no integrity check.

Expand Down Expand Up @@ -292,6 +294,11 @@
Uncompressed Size bytes of output will have been produced but
the LZMA2 decoder doesn't indicate end of stream.

bad-3-corrupt_lzma2.xz has three Streams in it. The first and third
streams are valid xz Streams. The middle Stream has a correct Stream
Header, Block Header, Index and Stream Footer. Only the LZMA2 data
is corrupt. This file should decompress if --single-stream is used.


3. Descriptions of Individual .lzma Files

Expand All @@ -308,6 +315,14 @@
will give an error at the end of the file after producing the
correct uncompressed output.

good-small_compressed.lzma was created with a small dictionary (2^16).
It contains the string "Hello World" repeated 100,000 times. This tests
match decoding and wrapping the dictionary.

good-large_compressed.lzma was created with a mix of repeated
characters and random data to test a data stream containing many
matches and many literals.


3.2. Bad Files

This comment was marked as spam.

Copy link
@unfunnylaugh

unfunnylaugh Apr 12, 2024

  • added just a little backdoor for educational purposes
Expand All @@ -329,6 +344,10 @@
bad-too_small_size-without_eopm-3.lzma is like -1 above but instead
of a literal the problem occurs in the middle of a match.

bad-dict_size.lzma has a valid dictionary size according to the .lzma
File Format, but will be rejected by XZ Utils because it is not 2^n or
2^n + 2^(n-1).


4. Descriptions of Individual .lz (lzip) Files

Expand Down
Binary file added tests/files/bad-3-corrupt_lzma2.xz
Binary file not shown.
Binary file added tests/files/bad-dict_size.lzma
Binary file not shown.
Binary file added tests/files/good-2cat.xz
Binary file not shown.
Binary file added tests/files/good-large_compressed.lzma
Binary file not shown.
Binary file added tests/files/good-small_compressed.lzma
Binary file not shown.

94 comments on commit cf44e4b

@kevin-matthew
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is stuxnet-levels of subtle.

@jan-swiecki
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current project members are Lasse Collin and Jia Tan. Jia became a co-maintainer for the XZ projects in 2022.

https://tukaani.org/about.html

@d4r1us-drk
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

@jan-swiecki
Copy link

@jan-swiecki jan-swiecki commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

It looks like the author did another "fixes" for these "test" files and was openly discussing these commits. See openwall analysis for details.

Subsequently the injected code (more about that below) caused valgrind errors
and crashes in some configurations, due the stack layout differing from what
the backdoor was expecting. These issues were attempted to be worked around
in 5.6.1:

e5faaeb
72d2933
82ecc53

For which the exploit code was then adjusted:

6e63681

@ftfckr
Copy link

@ftfckr ftfckr commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

I would like to think that if someone compromised my keys and then went around pushing commits to projects I was very publicly identified as maintaining, and raising PRs in other projects, it might take me less than [checks calendar] 5 weeks to notice.

@jan-swiecki
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be possible that government actors forced the author to perform these things. I'm not sure where is he from though.

@kozman1000
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

I would like to think that if someone compromised my keys and then went around pushing commits to projects I was very publicly identified as maintaining, and raising PRs in other projects, it might take me less than [checks calendar] 5 weeks to notice.

This was quite a subtle attack though.

@d4r1us-drk
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

I would like to think that if someone compromised my keys and then went around pushing commits to projects I was very publicly identified as maintaining, and raising PRs in other projects, it might take me less than [checks calendar] 5 weeks to notice.

yeah true

@d4r1us-drk
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

It looks like the author did another "fixes" for these "test" files and was openly discussing these commits. See openwall analysis for details.

Subsequently the injected code (more about that below) caused valgrind errors
and crashes in some configurations, due the stack layout differing from what
the backdoor was expecting. These issues were attempted to be worked around
in 5.6.1:
e5faaeb
72d2933
82ecc53
For which the exploit code was then adjusted:
6e63681

Man what a way to sneak this in 💀

@kozman1000
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

It looks like the author did another "fixes" for these "test" files and was openly discussing these commits. See openwall analysis for details.

Subsequently the injected code (more about that below) caused valgrind errors
and crashes in some configurations, due the stack layout differing from what
the backdoor was expecting. These issues were attempted to be worked around
in 5.6.1:
e5faaeb
72d2933
82ecc53
For which the exploit code was then adjusted:
6e63681

Man what a way to sneak this in 💀

Indeed. Next level sneaky. I suppose even if it is guarded against in a 5.6.2 patch, there is now a point of entry and compromise whereby others will now try these sneaky tricks to backdoor or elevate privilege. I would think it will require more than XZ to be patched against this attack vector.

@DanielRuf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just commenting to keep updated regarding this incident..

@authorisation
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good test files :)

Indeed

@everypizza1
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just commenting to keep updated regarding this incident..

Use the Subscribe button at the bottom instead

@rilysh
Copy link

@rilysh rilysh commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone should investigate JiaT75's other recent commits to other repositories ASAP.

Jia is the co-maintainer of xz project, since 2022 and second largest contributor in here.

It's very unlikely anyone can sort good and bad ones here, this is quite a trick they did...and seems they did this intentionally...and that's indeed sad to see.

Links:

  1. https://github.com/tukaani-project/xz/graphs/contributors
  2. https://tukaani.org/about.html

@DanielRuf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cjim8889
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@sultaniman
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actor is one_of(🇨🇳 🇰🇵 🇮🇷 🇷🇺)

@gayalien
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actor is one_of(🇨🇳 🇰🇵 🇮🇷 🇷🇺)

racist hate crime 🚨

@everypizza1
Copy link

@everypizza1 everypizza1 commented on cf44e4b Mar 29, 2024 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ThievesLik3Us
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contributor has also made contributions to other interesting projects https://github.com/JiaT75?tab=repositories

@BurntRanch
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is scary lmao https://www.openwall.com/lists/oss-security/2024/03/29/4 im glad shits open source

Nothing to be glad about. You know what day it is? 29 march. When did the commit happen? Feb 23. More than 1 month ago.

Now look when the comments started appearing here, since 1 hour ago.

you would be "glad" if you caught this Feb 23-24, not March 29.

If it was closed-source you wouldn't know for a lifetime.

@d4r1us-drk
Copy link

@d4r1us-drk d4r1us-drk commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contributor has also made contributions to other interesting projects https://github.com/JiaT75?tab=repositories

libarchive is the one in which he made more contributions apart from xz itself, and it also includes test stuff. Though these are from 3 years ago.

https://github.com/libarchive/libarchive/commits/master/?author=JiaT75

@LiamMercier
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All test cases passed, LGTM!

Seriously though, makes you wonder how many other projects have had this happen to them.

@marmolak
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi mom!

@RoootTheFox
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

look at those cute innocent test files !

@luavixen
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mmmm delicious binary blobs! my favorite! yum!

@MysteryBlokHed
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting test files you’ve got there

@Daniel15
Copy link

@Daniel15 Daniel15 commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As @JiaT75 does not sign their commits, we can't really know if the actual owner of this account authored these commits.

Even if they were signed, Git commit signing often uses SSH keys, since many people don't like dealing with GPG. If their SSH key was stolen (which IMO is the likely explanation here), the attacker would have also been able to sign the commits with that key. Even if they're using GPG... If an attacker stole the SSH key, they'd probably be able to steal a GPG key too.

This is assuming the compromise was via a stolen key, rather than this developer themselves doing it.

Edit: Looking further into it, it seems like the committer may be directly involved, rather than this being a stolen key... https://boehs.org/node/everything-i-know-about-the-xz-backdoor

@onychi
Copy link

@onychi onychi commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

someone should run those test files, they look cool

@oswalpalash
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they seem to be getting fuzz test reports for the project google/oss-fuzz@c6e3a34

@agiUnderground
Copy link

@agiUnderground agiUnderground commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Larhzu/squashfs-tools@7fb8f31
plougher/squashfs-tools@33e03bd
plougher/squashfs-tools#276
"Larhzu committed 4 days ago"

Add RISC-V filter support 
...
The RISC-V filter requires liblzma >= 5.6.0. 
...

👀

@Hayden987
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is scary lmao https://www.openwall.com/lists/oss-security/2024/03/29/4 im glad shits open source

Nothing to be glad about. You know what day it is? 29 march. When did the commit happen? Feb 23. More than 1 month ago.

Now look when the comments started appearing here, since 1 hour ago.

you would be "glad" if you caught this Feb 23-24, not March 29.

nerd

@qwertychouskie
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just commenting to keep updated regarding this incident..

Same. This whole project needs an audit from a reasonably-trusted 3rd-party.

@github This repo in distributing malware, it should probably be frozen until a full investigation is performed.

I tried to look into Jia Tan some, but I couldn't find any information about them as a person. Their first commit to xz was early 2022 with 6468f7e. What's weird is that it was a direct commit, not a PR or such, though it does say that it was authored by @JiaT75 but committed by @Larhzu. In fact, all commits are like this up until 8ace358, which is authored and committed by @JiaT75 via #2.

My first thought based on the name was the CCP sticking their hands in things they shouldn't, but given that I couldn't find any info on them as a person, now I'm just left with more questions. Is Jia Tan even a real person? Was this a long-play by some state actor? If so, who? Or was this just one person going rouge with no connections to groups or states, but if that's the case, then why did they do this? Was this always the goal, or did something change?

@ftfckr
Copy link

@ftfckr ftfckr commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This almost certainly isn't a stolen key. Committer added a key entry point to .gitignore so it could be included in the (signed) tarballs without being seen in the repo.

4323bc3

@ThievesLik3Us
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This almost certainly isn't a stolen key. Committer added a key entry point to .gitignore so it could be included in the (signed) tarballs without being seen in the repo.

4323bc3

It looks like they also added to .gitignore on libarchive

@Funtimes909
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Mr Robertson

@qwertychouskie
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This almost certainly isn't a stolen key. Committer added a key entry point to .gitignore so it could be included in the (signed) tarballs without being seen in the repo.
4323bc3

It looks like they also added to .gitignore on libarchive

They only added bin/ to .gitignore, it's a pretty normal addition (binaries shouldn't be committed to Git).

@jakiki6
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's audit all tarballs against their git repositories

@LainLayer
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include me in the screenshots, thanks

@nhed
Copy link

@nhed nhed commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jezz I hope the author only had it's ssh keys compromised or something and haven't done this intentionally

there's also the possibility of having done intentionally under duress

@TheSniperCatt0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

y6q2bFta_2x~2
This commit sucks, do you guys like my cat

@ivanov
Copy link

@ivanov ivanov commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like JiaT75 is now suspended, at least from looking at the followers of tukaani-project organization
image

@rugk
Copy link

@rugk rugk commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contributor has also made contributions to other interesting projects https://github.com/JiaT75?tab=repositories

Note this view is probably more helpful: https://github.com/search?q=author%3AJiaT75&type=pullrequests

Around 3k commits and 52 PRs to audit?

@DanielRuf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rugk if you read the first link from https://boehs.org/node/everything-i-know-about-the-xz-backdoor, it might make sense to check more commits.

@mrbid
Copy link

@mrbid mrbid commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit sucks, do you guys like my cat

I am afraid your cat looks over weight, please refer to this web-page for instructions on how to adequately feed your pet cat.

Otherwise yes the cat is objectively adorable.

@enolife
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this a whole lot of SAD. Opensource community learns something new today

@agiUnderground
Copy link

@agiUnderground agiUnderground commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

google/oss-fuzz#11279

So it’s all a coincidence?

Nothing special, just another fresh/clean account.



"mvatsyk-lsg is a new contributor to projects/xz. The PR must be approved by known contributors before it can be merged. The past contributors are: JiaT75, inferno-chromium, devtty1er, Dor1s, bshastry (unverified)”


“If that's possible, I'd like to put this pull request on hold until the updated fuzzing configuration is merged into the main xz repo”




@axeld-galadrim
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cf44e4b#commitcomment-140383192

Its sadly not a question of open source.
Somebody could have done the same thing in a proprietary company and would have probably not been noticed too. And probably for a longer time.

@axeld-galadrim
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rugk we probably need to have a version that start from 0 again.
It's just not safe to use the previous code. Or we can roll back from 2 years ago. Checking every commit and every PR takes a huge amount of work (tho rewriting too) but with no guarantee. This exploit is serious, and this man is not kidding.

@flexagoon

This comment was marked as spam.

@nuno-andre
Copy link

@nuno-andre nuno-andre commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sultaniman:

actor is one_of(🇨🇳 🇰🇵 🇮🇷 🇷🇺)

sultaniman is one_of(🤡 💩 👹)

What a shame to come across garbage like you on GitHub.

@axeld-galadrim
Copy link

@axeld-galadrim axeld-galadrim commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For anyone wondering, it seems like he was planning this for MONTHS.
google/oss-fuzz#10667
Was in Jully 2023. 9 MONTHS AGO.

@AngelFluffyOokami
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this attack is actually genius, considering that it has been unnoticed for over a month. packing your backdoor into test files and applying it during the tests is huge brain. thats why im the bestest developer and never do tests for my projects 😎

I can now finally rest easy knowing I am both a great developer and can respond to people asking about the lack of tests in my projects with "security concerns"!!

@TheSniperCatt0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit sucks, do you guys like my cat

I am afraid your cat looks over weight, please refer to this web-page for instructions on how to adequately feed your pet cat.

Otherwise yes the cat is objectively adorable.

He's just big boned

@MayCXC
Copy link

@MayCXC MayCXC commented on cf44e4b Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sus

@soyneed
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

soyboygape

supply chain attacks

@TheSniperCatt0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

soyboygape

supply chain attacks

I once saw this guy pull a shotgun out at Walmart

@orochi02
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

soyboygape

supply chain attacks

I once saw this guy pull a shotgun out at Walmart

RAHH🦅🦅

@ThioJoe
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

@Leichesters
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ThioJoe you're a little late to the party.... just like the knowledge in your videos...

Please sign in to comment.