Giteabot account was compromised #4167

lafriks · Jun 7, 2018

We are currently investigating suspiscious activity from an account with write access priviledge to go-gitea organization. A binary was added to releases across multiple go-gitea repositories. We cleaned up all releases and drop temporarily access from the account. We will investigate futhermore to understand what really happen to prevent it in the future and be transparent with you trough the process. In the meantime, if you find any suspicious activity please report them under this issue.

UPDATE: No source code or other Gitea infrastructure was affected, including https://dl.gitea.io/ so it's safe to use it to download Gitea binaries.

GitHub account that was hacked is under full control and also have set 2FA so this should not happen in future again.

What was done:

Most of go-gitea organization repositories new release&tag was created with name 0 and added install.exe binary (13KB in size) to that release that was malicious (from our analysis contained crypto currency miner). All these releases and binaries was deleted within 2-3 hours from when they were added.
Also 1.4.2 release windows Gitea .exe binary on GitHub was replaced by same 13K binary as above. So if Gitea is working, you are safe.
Just in case we did retag 1.4.2 to trigger CI to rebuild binaries so sha256 will be different now as it was before retag.

We have contacted GitHub but have not received any answer from them, yet

UPDATE2:
No actual gitea binaries were compromised so all hashes mentioned in comments below are safe.

SHA256 of malicious .exe file:
bfc5a0358b1ad76ffbc1e1f4670bd3240536e2fbac88272cee3003322a15fffe

UPDATE3:
v1.4.2 has been rereleased at about 2018-06-07 20:00:00 UTC+08

In the meantime please be careful when downloading our released binaries, especially the windows ones, until further notice. E.g. keep an eye on the size of the binaries, or a double .exe file ending.
We've found out our .exe binaries within release 1.4.2 were altered as well.

We work hard to follow all trails, clean up and get back to daily business as soon as possible.

daviian · Jun 7, 2018

In the meantime please be careful when downloading our released binaries, especially the windows ones, until further notice. E.g. keep an eye on the size of the binaries, or a double .exe file ending.
We've found out our .exe binaries within release 1.4.2 were altered as well.

We work hard to follow all trails, clean up and get back to daily business as soon as possible.

@daviian Maybe then it is necessary to sign releases?

Mauladen · Jun 7, 2018

@daviian Maybe then it is necessary to sign releases?

@Mauladen Thank you for your input. We definitely will discuss this.

daviian · Jun 7, 2018

@Mauladen Thank you for your input. We definitely will discuss this.

Ouch! Yeah, signed binaries would be ideal.

54 · Jun 7, 2018

Ouch! Yeah, signed binaries would be ideal.

Mauladen · Jun 7, 2018

@Mauladen We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones.

Or did you mean something else?

daviian · Jun 7, 2018

@Mauladen We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones.

Or did you mean something else?

This is normal. We retaged 1.4.2 to retrigger CI to rebuild binaries as windows binary for that release was removed and there was added new one malicious one

lafriks · Jun 7, 2018

This is normal. We retaged 1.4.2 to retrigger CI to rebuild binaries as windows binary for that release was removed and there was added new one malicious one

@daviian, Maybe @Mauladen meant that 1.4.2 release commit without GPG signature, but 1.4.1 was

axifive · Jun 7, 2018

@daviian, Maybe @Mauladen meant that 1.4.2 release commit without GPG signature, but 1.4.1 was

Still need to edit the list of changes

Mauladen · Jun 7, 2018

Still need to edit the list of changes

@axifive commit where not gpg signed by user but github that does it automaticaly (with github key) when the merger is the same the PR author. I wouldn't it consider more safer because if github account were compromised I will been also show as "verified". But we could start to sign tag from now (to be discuss). For information we are, working on signing binary since it is more easy to corrumpt thzan binary as the git commit tree.

sapk · Jun 7, 2018

@axifive commit where not gpg signed by user but github that does it automaticaly (with github key) when the merger is the same the PR author. I wouldn't it consider more safer because if github account were compromised I will been also show as "verified". But we could start to sign tag from now (to be discuss). For information we are, working on signing binary since it is more easy to corrumpt thzan binary as the git commit tree.

You should upload a tarball created by git-archive (in addition to those that github automatically generates) which you sign with signify. You can get an idea on how that works/looks here:

Libressl uses the same technology.

hasufell · Jun 7, 2018

You should upload a tarball created by git-archive (in addition to those that github automatically generates) which you sign with signify. You can get an idea on how that works/looks here:

Libressl uses the same technology.

Is there anymore information on this? What was the payload of the binary? Are users going to be informed about this through a blog post or anything? Were any of the linux binaries compromised? I'm rather concerned about what these things may have done to my servers. I'm doubly concerned that I only found out about this browsing the issue page by chance vs an official notice on the project page/blog

CountMurphy · Jun 7, 2018

Is there anymore information on this? What was the payload of the binary? Are users going to be informed about this through a blog post or anything? Were any of the linux binaries compromised? I'm rather concerned about what these things may have done to my servers. I'm doubly concerned that I only found out about this browsing the issue page by chance vs an official notice on the project page/blog

Is the Gitea 1.4.2 release safe to update to from source code at this point in time?

asiekierka · Jun 7, 2018

Is the Gitea 1.4.2 release safe to update to from source code at this point in time?

At this point, it's not clear to me whether only binaries were affected. How did you verify this? Are your branches protected?

hasufell · Jun 7, 2018

At this point, it's not clear to me whether only binaries were affected. How did you verify this? Are your branches protected?

shasums differ on binaries I downloaded on 6/4 and 6/7 though they are the same number of bytes:

$ sha256sum gitea-1.4.2-linux-amd64*
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d  gitea-1.4.2-linux-amd64.1

$ ls --color=tty -l gitea-1.4.2-linux-amd64*
-rwxr-xr-x 1 matt matt 53674800 Jun  4 20:39 gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 matt matt 53674800 Jun  7 08:18 gitea-1.4.2-linux-amd64.1

xcolour · Jun 7, 2018

shasums differ on binaries I downloaded on 6/4 and 6/7 though they are the same number of bytes:

$ sha256sum gitea-1.4.2-linux-amd64*
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d  gitea-1.4.2-linux-amd64.1

$ ls --color=tty -l gitea-1.4.2-linux-amd64*
-rwxr-xr-x 1 matt matt 53674800 Jun  4 20:39 gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 matt matt 53674800 Jun  7 08:18 gitea-1.4.2-linux-amd64.1

Feel like hosting those somewhere so folks can tear them apart?

bkero · Jun 7, 2018

Feel like hosting those somewhere so folks can tear them apart?

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA. I'll put them somewhere more permanent if necessary.

xcolour · Jun 7, 2018

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA. I'll put them somewhere more permanent if necessary.

Question - is it just the binaries on the website that were affected, or were the repositories also affected? I ask because yesterday I heard about Gitea and cloned and built the v1.4 branch.

jatherrien · Jun 7, 2018

Question - is it just the binaries on the website that were affected, or were the repositories also affected? I ask because yesterday I heard about Gitea and cloned and built the v1.4 branch.

Would really like some more info if the source itself is compromised or just the binaries. I run a script to check for updates/build from git nightly and was lucky enough to miss this because of the timing.

Is the current 1.4.2 release verified to be clean? If we know that's tainted we should pull that ASAP and put it somewhere else for analysis, otherwise people will still be downloading that if they don't see this issue. I agree with @CountMurphy, can we add something to the README so it's really visible?

4oo4 · Jun 7, 2018

Would really like some more info if the source itself is compromised or just the binaries. I run a script to check for updates/build from git nightly and was lucky enough to miss this because of the timing.

Is the current 1.4.2 release verified to be clean? If we know that's tainted we should pull that ASAP and put it somewhere else for analysis, otherwise people will still be downloading that if they don't see this issue. I agree with @CountMurphy, can we add something to the README so it's really visible?

Can we get hashes so we can check if our binaries are affected? My gitea 1.4.1 (linux amd64) looks like this:
$ sha256sum gitea
d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 gitea

mikedilger · Jun 7, 2018

Can we get hashes so we can check if our binaries are affected? My gitea 1.4.1 (linux amd64) looks like this:
$ sha256sum gitea
d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 gitea

I just downloaded gitea 1.4.1 linux-amd-64 and I got d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 as the sha256 too

cupnoodle · Jun 7, 2018

I just downloaded gitea 1.4.1 linux-amd-64 and I got d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 as the sha256 too

@mikedilger See: #4167 (comment)

4oo4 · Jun 7, 2018

@mikedilger See: #4167 (comment)

To make it overwhelmingly clear: nobody knows anything and the only safe hashes are the ones currently found here: https://github.com/go-gitea/gitea/releases

For Gitea 1.4.2 on amd64, that means:

c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d

Sorry, I misunderstood #4167 (comment) to mean that both binaries were compromised.

I've edited this post to remove any ambiguities about what we actually know.

christianbundy · Jun 7, 2018

To make it overwhelmingly clear: nobody knows anything and the only safe hashes are the ones currently found here: https://github.com/go-gitea/gitea/releases

For Gitea 1.4.2 on amd64, that means:

c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d

Sorry, I misunderstood #4167 (comment) to mean that both binaries were compromised.

I've edited this post to remove any ambiguities about what we actually know.

Hold on: according to this comment (#4167 (comment)), there are two shas e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 4th and c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d from June 7th.

Are we saying that both of these are compromised or just one of them? For the record the one on my server is e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 5th.

shuhaowu · Jun 7, 2018

Hold on: according to this comment (#4167 (comment)), there are two shas e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 4th and c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d from June 7th.

Are we saying that both of these are compromised or just one of them? For the record the one on my server is e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 5th.

@christianbundy, Please, don't make hasty conclusions, wait for a response from the Team member

axifive · Jun 7, 2018

@christianbundy, Please, don't make hasty conclusions, wait for a response from the Team member

Please, don't make hasty conclusions, wait for a response from the Team member

Sorry about that, I thought the file hashes would be posted by a team member immediately and didn't realize that the info was coming from someone else who was also in the dark. Is this the correct channel to watch for the latest developments? I've powered off my server and need more information to tell whether I've been infected.

christianbundy · Jun 7, 2018

Please, don't make hasty conclusions, wait for a response from the Team member

Sorry about that, I thought the file hashes would be posted by a team member immediately and didn't realize that the info was coming from someone else who was also in the dark. Is this the correct channel to watch for the latest developments? I've powered off my server and need more information to tell whether I've been infected.

I see your edits crossing out the entry I pointed to. HOWEVER, isn't this too early to draw conclusion from? How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

We're still missing a couple pieces of information (likely non-exhaustive):

Which binaries were compromised and what are their checksums?
When did the bot account get compromised?
Is the source repository compromised (this might be easy to check if you have a version of the repo cloned from some while back, then you can maybe check diffs or evidence of force pushes).

shuhaowu · Jun 7, 2018

I see your edits crossing out the entry I pointed to. HOWEVER, isn't this too early to draw conclusion from? How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

We're still missing a couple pieces of information (likely non-exhaustive):

Which binaries were compromised and what are their checksums?
When did the bot account get compromised?
Is the source repository compromised (this might be easy to check if you have a version of the repo cloned from some while back, then you can maybe check diffs or evidence of force pushes).

I got:

# ls -la gitea-1.4.1-linux-amd64
-rwxr-xr-x 1 gitea gitea 53663640 May  3 08:02 gitea-1.4.1-linux-amd64

With sha256sum d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851

and I just downloaded gitea-1.4.2-linux-amd64:

# ls -la gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 root root 53674800 Jun  7 14:18 gitea-1.4.2-linux-amd64

With sha256sum c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d which matches the provided .sha256 file: gitea-1.4.2-linux-amd64: OK which should be safe. (right?)

[...] We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones. [...]

I uploaded gitea-1.4.2-linux-amd64 for analysis here, but it doesn't really tell anything obvious.

Going to watch this issue and keep my gitea installation offline for the time being.

vmp32k · Jun 7, 2018

I got:

# ls -la gitea-1.4.1-linux-amd64
-rwxr-xr-x 1 gitea gitea 53663640 May  3 08:02 gitea-1.4.1-linux-amd64

With sha256sum d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851

and I just downloaded gitea-1.4.2-linux-amd64:

# ls -la gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 root root 53674800 Jun  7 14:18 gitea-1.4.2-linux-amd64

With sha256sum c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d which matches the provided .sha256 file: gitea-1.4.2-linux-amd64: OK which should be safe. (right?)

[...] We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones. [...]

I uploaded gitea-1.4.2-linux-amd64 for analysis here, but it doesn't really tell anything obvious.

Going to watch this issue and keep my gitea installation offline for the time being.

How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

@shuhaowu I said that c843d462 was fine, not e14e54f3. We know this because that's the file currently being served on GitHub, which they have recently rebuilt.

christianbundy · Jun 7, 2018

How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

@shuhaowu I said that c843d462 was fine, not e14e54f3. We know this because that's the file currently being served on GitHub, which they have recently rebuilt.

If 1.4.2 was rebuilt, that should be signed and verified like the other valid releases. Not doing so only adds to the confusion.

4oo4 · Jun 7, 2018

If 1.4.2 was rebuilt, that should be signed and verified like the other valid releases. Not doing so only adds to the confusion.

@xcolour

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA

The only difference between these two binaries is that about 2000 instances of movabsq $63663754793, %rcx were replaced with movabsq $63663969431, %rcx. I can't say exactly what that changes behaviorally, but I think it's very unlikely that it's enough to be malicious.

spaghetti2514 · Jun 7, 2018

@xcolour

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA

The only difference between these two binaries is that about 2000 instances of movabsq $63663754793, %rcx were replaced with movabsq $63663969431, %rcx. I can't say exactly what that changes behaviorally, but I think it's very unlikely that it's enough to be malicious.

Maybe push out a new (signed?) 1.4.3 release with the known-safe code, or would that confuse things more?

justinclift · Jun 7, 2018

Maybe push out a new (signed?) 1.4.3 release with the known-safe code, or would that confuse things more?

@justinclift I like that idea, also a blog post and something in the README about the situation would help clear things up.

4oo4 · Jun 7, 2018

@justinclift I like that idea, also a blog post and something in the README about the situation would help clear things up.

@spaghetti2514

On one hand, you're right -- the differences between the those two binaries is easy to see.

On the other hand, the Gitea team hasn't actually posted which binaries were affected, posted any SHA256 hashes, or really said anything that would help us understand the compromise As others have pointed out, we don't have nearly enough info to determine what happened and who's affected.

I'm frustrated to point out that we should probably just assume the worst and wait for diagnostic steps from the core team. With that said, I appreciate you posting the disassembled diff.

christianbundy · Jun 7, 2018

@spaghetti2514

On one hand, you're right -- the differences between the those two binaries is easy to see.

On the other hand, the Gitea team hasn't actually posted which binaries were affected, posted any SHA256 hashes, or really said anything that would help us understand the compromise As others have pointed out, we don't have nearly enough info to determine what happened and who's affected.

I'm frustrated to point out that we should probably just assume the worst and wait for diagnostic steps from the core team. With that said, I appreciate you posting the disassembled diff.

Is this what you're looking for? #4167 (comment)

saagarjha · Jun 7, 2018

Is this what you're looking for? #4167 (comment)

Thank you for the update @lafriks !

CountMurphy · Jun 7, 2018

Thank you for the update @lafriks !

I have updated issue description with additional info. It was not so bad as it could have been. We wanted to be as transparent as possible so created this issue as soon as we did clean up & fixed everything and as this was first time something like this have happened, so probably we should have given more information faster, sorry for causing confusion.

lafriks · Jun 7, 2018

I have updated issue description with additional info. It was not so bad as it could have been. We wanted to be as transparent as possible so created this issue as soon as we did clean up & fixed everything and as this was first time something like this have happened, so probably we should have given more information faster, sorry for causing confusion.

@lafriks Thanks for the update! Could you post your PGP key that we can use going forward to verify commits?

4oo4 · Jun 7, 2018

@lafriks Thanks for the update! Could you post your PGP key that we can use going forward to verify commits?

@rugk Useful looking script. 😄

justinclift · Jun 8, 2018

@rugk Useful looking script. 😄

Has any contributing team member got an archive of 1.4.2 prior to breach discovery?

You seem to have left a lot of people in the dark on whether or not this binary has been altered for any Linux releases, which is quite an important piece of information considering:

Most deployments will be to Linux stacks
Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

While I'd like to believe that this was a simple Windows-targetted attack and nothing more, the fact that they targeted this specific repository just days into a dramatic spike in userbase growth seems indicative to a more well orchestrated effort. I highly doubt that anyone who knew what they were targeting and how to pull off this sort of thing would have just passed up such a ripe opportunity to infect as many hosts as possible.

stanier · Jun 8, 2018

Has any contributing team member got an archive of 1.4.2 prior to breach discovery?

You seem to have left a lot of people in the dark on whether or not this binary has been altered for any Linux releases, which is quite an important piece of information considering:

Most deployments will be to Linux stacks
Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

While I'd like to believe that this was a simple Windows-targetted attack and nothing more, the fact that they targeted this specific repository just days into a dramatic spike in userbase growth seems indicative to a more well orchestrated effort. I highly doubt that anyone who knew what they were targeting and how to pull off this sort of thing would have just passed up such a ripe opportunity to infect as many hosts as possible.

@stanier dl.gitea.io was not affected and we have verified that no other binaries were tampered with

lafriks · Jun 8, 2018

@stanier dl.gitea.io was not affected and we have verified that no other binaries were tampered with

So that explains why our webproxy refused to download the latest gitea image from hub.docker.com ...

hhenkel · Jun 8, 2018

So that explains why our webproxy refused to download the latest gitea image from hub.docker.com ...

@lafriks So can you confirm that c843d462 and e14e54f3 are both safe? If the CI provided a build with a different signature, then something must have changed in either the source code or the parameters the compiler used for the build. It's a minor technicality by itself, but it was never directly stated here whether or not the adversary had altered the 1.4.2 Linux binaries, only the respective .exe binaries mentioned in the comment by @daviian.

Just to clarify, I'm asking about the GH repo's release page binaries, not dl.gitea.io, I understand nothing was tampered with outside of the GH repo.

stanier · Jun 8, 2018

@lafriks So can you confirm that c843d462 and e14e54f3 are both safe? If the CI provided a build with a different signature, then something must have changed in either the source code or the parameters the compiler used for the build. It's a minor technicality by itself, but it was never directly stated here whether or not the adversary had altered the 1.4.2 Linux binaries, only the respective .exe binaries mentioned in the comment by @daviian.

Just to clarify, I'm asking about the GH repo's release page binaries, not dl.gitea.io, I understand nothing was tampered with outside of the GH repo.

Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

Hmmm, don't most package managers (and similar) have provision for verifying at least sha* checksums? Not necessarily just for detecting malware, but as a "lets verify the download wasn't corrupt".

justinclift · Jun 8, 2018

Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

Hmmm, don't most package managers (and similar) have provision for verifying at least sha* checksums? Not necessarily just for detecting malware, but as a "lets verify the download wasn't corrupt".

@justinclift yes, but that only applies to binaries installed through the package manager. The issue in question here concerns with a direct download from GitHub, which doesn't have any embedded malware detection to the best of my knowledge.

stanier · Jun 8, 2018

@justinclift yes, but that only applies to binaries installed through the package manager. The issue in question here concerns with a direct download from GitHub, which doesn't have any embedded malware detection to the best of my knowledge.

Ahhh. The term "Linux stacks" threw me off, as I'm more used to direct downloads being simplistic one-off things (eg when prototyping), rather than something an automated solution would do. No worries. 😄

justinclift · Jun 8, 2018

Ahhh. The term "Linux stacks" threw me off, as I'm more used to direct downloads being simplistic one-off things (eg when prototyping), rather than something an automated solution would do. No worries. 😄

@stanier the build was redone by drone to clear all. Go doesn't provide same binary each time you build the binary since some variable could change (like build time) so it is normal that the hash differs.
I got all binary during investigation and can provide hashs as soon as I get to my personal computer.

For all, please stop argument on rumors and provide constructive input.
I understand that you worry for your security and we care greatly about it (since we are also at your place as user of gitea). Currently the access were changed and we haven't seen any suspicious activity anymore. We have done this issue to inform as soon as possible to be transparent and be able to receive input from any data usefull to investigate or missed spot as no one is perfect.
We will do a post-mortem to explain what happen and we have definitively think of action to prevent this in the futur.
If this issue go wild, I think we will need to close to keep only usefull and concise information and send the debate to discord where it should go.

sapk · Jun 8, 2018

@stanier the build was redone by drone to clear all. Go doesn't provide same binary each time you build the binary since some variable could change (like build time) so it is normal that the hash differs.
I got all binary during investigation and can provide hashs as soon as I get to my personal computer.

For all, please stop argument on rumors and provide constructive input.
I understand that you worry for your security and we care greatly about it (since we are also at your place as user of gitea). Currently the access were changed and we haven't seen any suspicious activity anymore. We have done this issue to inform as soon as possible to be transparent and be able to receive input from any data usefull to investigate or missed spot as no one is perfect.
We will do a post-mortem to explain what happen and we have definitively think of action to prevent this in the futur.
If this issue go wild, I think we will need to close to keep only usefull and concise information and send the debate to discord where it should go.

To avoid a situation like that in the future we will start to sign all binaries with the next version. I have built a plugin for Drone which you can find at https://github.com/drone-plugins/drone-gpgsign (the documentation for this plugin have to be done) that will sign all binaries, the public key will be uploaded to the download server and to a keyserver, than you will always be able to verify the binaries in a trustable way.

Just to show you an example how this will look and what are the results you can take a look at https://github.dronehippie.de/webhippie/ldap-proxy/49/18 and https://dl.webhippie.de/misc/ldap-proxy/master/, similar files will be uploaded to the Gitea download page and to the GitHub releases.

If you think this plugin is missing something really important, feel free to open an issue on the plugin repository and I can address it.

tboerger · Jun 8, 2018

To avoid a situation like that in the future we will start to sign all binaries with the next version. I have built a plugin for Drone which you can find at https://github.com/drone-plugins/drone-gpgsign (the documentation for this plugin have to be done) that will sign all binaries, the public key will be uploaded to the download server and to a keyserver, than you will always be able to verify the binaries in a trustable way.

Just to show you an example how this will look and what are the results you can take a look at https://github.dronehippie.de/webhippie/ldap-proxy/49/18 and https://dl.webhippie.de/misc/ldap-proxy/master/, similar files will be uploaded to the Gitea download page and to the GitHub releases.

If you think this plugin is missing something really important, feel free to open an issue on the plugin repository and I can address it.

@sapk Thanks for the clarification. I apologize, it completely slipped my mind that the project was Golang-based-- most issues like this deal with older software so I think my mind jumped to a false relation. My mistake.

stanier · Jun 8, 2018

@sapk Thanks for the clarification. I apologize, it completely slipped my mind that the project was Golang-based-- most issues like this deal with older software so I think my mind jumped to a false relation. My mistake.

I've started to take a look at the install.exe binary that was uploaded: https://grh.am/2018/a-look-at-the-compromised-gitea-release/

It seems it wasn't just Gitea that got hit by this, but also https://github.com/opencompany/www.opencompany.org/releases

graystevens · Jun 8, 2018

I've started to take a look at the install.exe binary that was uploaded: https://grh.am/2018/a-look-at-the-compromised-gitea-release/

It seems it wasn't just Gitea that got hit by this, but also https://github.com/opencompany/www.opencompany.org/releases

Thank you for the clear explanation.

I think this issue is the main reason to go through distro-specific packaging. It brings more attention to the packaging and potential malicious code/binary (especially in case of such a gross attempt). It would be great to have at least Debian/RedHat/Archlinux packages for Gitea : that would prevent many people from running an arbitrary binary on their production server :)

Is PGP-signing the releases enough? Probably. Just make sure to advertise your public key on many different platforms so that compromising for instance your website does not make everybody download wrong keys. (But a Debian package in the backports would be <3)

(Also, reproducible builds ?)

paulcmal · Jun 8, 2018

Thank you for the clear explanation.

I think this issue is the main reason to go through distro-specific packaging. It brings more attention to the packaging and potential malicious code/binary (especially in case of such a gross attempt). It would be great to have at least Debian/RedHat/Archlinux packages for Gitea : that would prevent many people from running an arbitrary binary on their production server :)

Is PGP-signing the releases enough? Probably. Just make sure to advertise your public key on many different platforms so that compromising for instance your website does not make everybody download wrong keys. (But a Debian package in the backports would be <3)

(Also, reproducible builds ?)

Since you will start signing binary releases, could you also consider signing the Docker images pushed to the registry?

vbrandl · Jun 8, 2018

Since you will start signing binary releases, could you also consider signing the Docker images pushed to the registry?

It seems it wasn't just Gitea that got hit by this, but also https://github.com/opencompany/www.opencompany.org/releases

Just emailed their contact people directly, in case they're not looking at their GitHub issues yet.

justinclift · Jun 8, 2018

It seems it wasn't just Gitea that got hit by this, but also https://github.com/opencompany/www.opencompany.org/releases

Just emailed their contact people directly, in case they're not looking at their GitHub issues yet.

@graystevens Should the pastebin staff be contacted to kill that pastebin address, or is it better to fully analyse the malware first so it's properly understood?

justinclift · Jun 8, 2018

@graystevens Should the pastebin staff be contacted to kill that pastebin address, or is it better to fully analyse the malware first so it's properly understood?

Thanks everyone.. Would re download and put my server back up

adelowo · Jun 8, 2018

Thanks everyone.. Would re download and put my server back up

@justinclift I’ll report the paste to PasteBin now - I’ve got a copy of the contents so we can recreate the output for the malware should we need to. Good shout

graystevens · Jun 8, 2018

@justinclift I’ll report the paste to PasteBin now - I’ve got a copy of the contents so we can recreate the output for the malware should we need to. Good shout

To be fair, go is now has reproducible build : https://blog.filippo.io/reproducing-go-binaries-byte-by-byte/
That maybe cgo for sqlite and some build env that make them not reproducible.

sapk · Jun 8, 2018

To be fair, go is now has reproducible build : https://blog.filippo.io/reproducing-go-binaries-byte-by-byte/
That maybe cgo for sqlite and some build env that make them not reproducible.

Just for information, the previous rebuild and un-touched hash list. If you have those hash it means you have the binary before the rebuild we have done to reset the 1.4.2 release and also before the attack. If it is the case, you can stay with them.

156655506d373241fea6b8955acbe6fdc148f06b49b4f6e46d11cadcd00d7316  gitea-1.4.2-darwin-10.6-386
5730c1a471dfc2c055442360d431c950b3a4f266403c5f327c94a68b88e67a10  gitea-1.4.2-darwin-10.6-amd64
8c3cc2127161695dea512176cd4282711e8c57972f333253cc89597dfab002ef  gitea-1.4.2-linux-386
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
bca75d81c52b9aa57fd05b8f7ce0572abae3e1a008d8ab77840ca08c53975867  gitea-1.4.2-linux-arm-5
3aa791afce2e3450461038e09622fe04f34b891c47db641be50b6cc3f772645e  gitea-1.4.2-linux-arm64
fc73d06621fc23a944a2a8ee3b19fbd8edb972b0cdaefa67248eb885aa4d6240  gitea-1.4.2-linux-arm-6
5b76cd9b84846c09ba3627219a6cad99542a53d8eec71a427a433ab82eaabacf  gitea-1.4.2-linux-arm-7
4fafeabfa069066fd624364b47b5729f8f068545d4cd8a3620774a982937ac16  gitea-1.4.2-linux-mips64le
7d9e0afcd315f0efbfb26cab303b3fee3939fa0e081b0d3f4f480f950d0a9870  gitea-1.4.2-linux-mips64
7f2e3c1163823889bec1fdd34b4135149c83d53b6dc86f186821e51e0f839e53  gitea-1.4.2-linux-mipsle
a1b8338113d4fdb0360582ba18f6fce97722c779493e7d7e07594d78c7c3f003  gitea-1.4.2-linux-mips
82ad50932bd927ead2e7da4e4c575d94f88e0105a82eda4cce1df7c9fc5ba0dc  gitea-1.4.2-windows-4.0-386.exe
86d7332894390ccaedd39be891044d829f54d4cd71fa21fce5dbd9bc3abbce44  gitea-1.4.2-windows-4.0-amd64.exe

sapk · Jun 8, 2018

Just for information, the previous rebuild and un-touched hash list. If you have those hash it means you have the binary before the rebuild we have done to reset the 1.4.2 release and also before the attack. If it is the case, you can stay with them.

156655506d373241fea6b8955acbe6fdc148f06b49b4f6e46d11cadcd00d7316  gitea-1.4.2-darwin-10.6-386
5730c1a471dfc2c055442360d431c950b3a4f266403c5f327c94a68b88e67a10  gitea-1.4.2-darwin-10.6-amd64
8c3cc2127161695dea512176cd4282711e8c57972f333253cc89597dfab002ef  gitea-1.4.2-linux-386
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
bca75d81c52b9aa57fd05b8f7ce0572abae3e1a008d8ab77840ca08c53975867  gitea-1.4.2-linux-arm-5
3aa791afce2e3450461038e09622fe04f34b891c47db641be50b6cc3f772645e  gitea-1.4.2-linux-arm64
fc73d06621fc23a944a2a8ee3b19fbd8edb972b0cdaefa67248eb885aa4d6240  gitea-1.4.2-linux-arm-6
5b76cd9b84846c09ba3627219a6cad99542a53d8eec71a427a433ab82eaabacf  gitea-1.4.2-linux-arm-7
4fafeabfa069066fd624364b47b5729f8f068545d4cd8a3620774a982937ac16  gitea-1.4.2-linux-mips64le
7d9e0afcd315f0efbfb26cab303b3fee3939fa0e081b0d3f4f480f950d0a9870  gitea-1.4.2-linux-mips64
7f2e3c1163823889bec1fdd34b4135149c83d53b6dc86f186821e51e0f839e53  gitea-1.4.2-linux-mipsle
a1b8338113d4fdb0360582ba18f6fce97722c779493e7d7e07594d78c7c3f003  gitea-1.4.2-linux-mips
82ad50932bd927ead2e7da4e4c575d94f88e0105a82eda4cce1df7c9fc5ba0dc  gitea-1.4.2-windows-4.0-386.exe
86d7332894390ccaedd39be891044d829f54d4cd71fa21fce5dbd9bc3abbce44  gitea-1.4.2-windows-4.0-amd64.exe

The install.exe cleanup pass seems to have missed quite a few repositories:

Most of these are old and not exactly relevant, but it's probably not a good idea to keep malware around.

jvanraaij · Jun 20, 2018

The install.exe cleanup pass seems to have missed quite a few repositories:

Most of these are old and not exactly relevant, but it's probably not a good idea to keep malware around.

@lunny

go-xorm

go-tango

go-xweb

goftp

gobook

tango

gobuild

yasuokav · Jun 20, 2018

@lunny

go-xorm

go-tango

go-xweb

goftp

gobook

tango

gobuild

Ugh, thanks. What's the approach you're using to locate these? It seems that whatever approach the GitHub staff have used hasn't really been 100% effective. 😦

justinclift · Jun 20, 2018

Ugh, thanks. What's the approach you're using to locate these? It seems that whatever approach the GitHub staff have used hasn't really been 100% effective. 😦

@jvanraaij @yaggytter thanks.

lunny · Jun 21, 2018

@jvanraaij @yaggytter thanks.

As after our investigation nothing else was affected and no more information was given by GitHub I think this issue can be closed. Anyone to write blog post about this?

lafriks · Jun 25, 2018

As after our investigation nothing else was affected and no more information was given by GitHub I think this issue can be closed. Anyone to write blog post about this?

@lafriks I posted this blog post back within a couple of days of this all kicking off - its more of a look at the malware than the issue at hand, but I'm happy to update it if you feel something is worth documenting 👍

graystevens · Jun 25, 2018

@lafriks I posted this blog post back within a couple of days of this all kicking off - its more of a look at the malware than the issue at hand, but I'm happy to update it if you feel something is worth documenting 👍

I think he wants to write a post-mortem blog post on the gitea blog :)

tboerger · Jun 25, 2018

I think he wants to write a post-mortem blog post on the gitea blog :)

@graystevens BTW your site notice link is a 404 one. 😄

rugk · Jun 25, 2018

@graystevens BTW your site notice link is a 404 one. 😄

As after our investigation nothing else was affected and no more information was given by GitHub ...

As a data point, although GitHub haven't said anything about this in public, privately they've responded (via email) to say effectively "Thanks, we're investigating."

The above comments by @jvanraaij and @yasuokav seemed to help too, as (weirdly, from my PoV) GitHub don't seem to have found those particular instances of the malware prior to that.

justinclift · Jun 25, 2018

As after our investigation nothing else was affected and no more information was given by GitHub ...

As a data point, although GitHub haven't said anything about this in public, privately they've responded (via email) to say effectively "Thanks, we're investigating."

The above comments by @jvanraaij and @yasuokav seemed to help too, as (weirdly, from my PoV) GitHub don't seem to have found those particular instances of the malware prior to that.

For example, all of the repo's listed by @yasuokav here still have the malware: crossoverJie/SSM#36

I'll email GitHub staff again to point it out. They seem to get things taken care of within a few days when contacted directly with an exact list to look at.

justinclift · Jun 25, 2018

For example, all of the repo's listed by @yasuokav here still have the malware: crossoverJie/SSM#36

I'll email GitHub staff again to point it out. They seem to get things taken care of within a few days when contacted directly with an exact list to look at.

This is sad for all the other projects, but at least for Gitea we have solved the issues properly, hopefully... :)

tboerger · Jun 25, 2018

This is sad for all the other projects, but at least for Gitea we have solved the issues properly, hopefully... :)

lafriks added kind/security priority/critical labels Jun 7, 2018

axifive referenced this issue in opencompany/www.opencompany.org Jun 8, 2018
Closed
Explain release #176

This was referenced Jun 8, 2018

Open

Provide reproductible binary (if possible) #4181

Closed

Provide a gpg signature of binaries #4182

This was referenced Jun 8, 2018

Merged

Add security advice to contrib guide #4187

Merged

Add gpg-sign step to drone #4188

yasuokav referenced this issue in crossoverJie/SSM Jun 21, 2018
Closed
Please check your account security #36

dR3b added a commit to dR3b/vagrant-gitea that referenced this issue Jun 22, 2018

change sha256: go-gitea/gitea#4167

f817ba1

mewmew referenced this issue in diasurgical/devilution Jun 28, 2018
Closed
Trojan in released Devilution.exe ??? #97

beerisgood referenced this issue in Telegram-FOSS-Team/Telegram-FOSS Jul 1, 2018
Open
Move from Github #248

go-gitea/gitea

Join GitHub today

Giteabot account was compromised #4167

Comments

lafriks commented Jun 7, 2018 • edited by lunny Edited 4 times lunny edited Jun 8, 2018 (most recent) lafriks edited Jun 8, 2018 lafriks edited Jun 7, 2018 lafriks edited Jun 7, 2018 lafriks created Jun 7, 2018

lafriks added kind/security priority/critical labels Jun 7, 2018

This comment has been minimized.

daviian commented Jun 7, 2018

This comment has been minimized.

Mauladen commented Jun 7, 2018

This comment has been minimized.

daviian commented Jun 7, 2018

This comment has been minimized.

54 commented Jun 7, 2018

This comment has been minimized.

Mauladen commented Jun 7, 2018

This comment has been minimized.

daviian commented Jun 7, 2018 • edited Edited 1 time daviian edited Jun 7, 2018 (most recent) daviian created Jun 7, 2018

This comment has been minimized.

lafriks commented Jun 7, 2018

This comment has been minimized.

This comment has been minimized.

Mauladen commented Jun 7, 2018

This comment has been minimized.

sapk commented Jun 7, 2018

This comment has been minimized.

hasufell commented Jun 7, 2018

This comment has been minimized.

CountMurphy commented Jun 7, 2018 • edited Edited 3 times CountMurphy edited Jun 7, 2018 (most recent) CountMurphy edited Jun 7, 2018 CountMurphy edited Jun 7, 2018 CountMurphy created Jun 7, 2018

This comment has been minimized.

asiekierka commented Jun 7, 2018

This comment has been minimized.

hasufell commented Jun 7, 2018

This comment has been minimized.

xcolour commented Jun 7, 2018

This comment has been minimized.

bkero commented Jun 7, 2018

This comment has been minimized.

xcolour commented Jun 7, 2018 • edited Edited 1 time xcolour edited Jun 7, 2018 (most recent) xcolour created Jun 7, 2018

This comment has been minimized.

jatherrien commented Jun 7, 2018

This comment has been minimized.

4oo4 commented Jun 7, 2018

This comment has been minimized.

mikedilger commented Jun 7, 2018 • edited Edited 1 time mikedilger edited Jun 7, 2018 (most recent) mikedilger created Jun 7, 2018

This comment has been minimized.

cupnoodle commented Jun 7, 2018

This comment has been minimized.

4oo4 commented Jun 7, 2018

This comment has been minimized.

christianbundy commented Jun 7, 2018 • edited Edited 2 times christianbundy edited Jun 7, 2018 (most recent) christianbundy edited Jun 7, 2018 christianbundy created Jun 7, 2018

This comment has been minimized.

shuhaowu commented Jun 7, 2018

This comment has been minimized.

axifive commented Jun 7, 2018

This comment has been minimized.

christianbundy commented Jun 7, 2018

This comment has been minimized.

shuhaowu commented Jun 7, 2018

This comment has been minimized.

vmp32k commented Jun 7, 2018 • edited Edited 1 time vmp32k edited Jun 7, 2018 (most recent) vmp32k created Jun 7, 2018

This comment has been minimized.

christianbundy commented Jun 7, 2018

This comment has been minimized.

4oo4 commented Jun 7, 2018

This comment has been minimized.

spaghetti2514 commented Jun 7, 2018

This comment has been minimized.

justinclift commented Jun 7, 2018

This comment has been minimized.

4oo4 commented Jun 7, 2018

This comment has been minimized.

christianbundy commented Jun 7, 2018 • edited Edited 1 time christianbundy edited Jun 7, 2018 (most recent) christianbundy created Jun 7, 2018

This comment has been minimized.

saagarjha commented Jun 7, 2018

This comment has been minimized.

CountMurphy commented Jun 7, 2018

This comment has been minimized.

lafriks commented Jun 7, 2018

This comment has been minimized.

lafriks commented Jun 7, 2018 •

edited by lunny

Edited 4 times

lunny edited Jun 8, 2018 (most recent)

lafriks edited Jun 8, 2018

lafriks edited Jun 7, 2018

lafriks edited Jun 7, 2018

lafriks created Jun 7, 2018

daviian commented Jun 7, 2018 •

edited

Edited 1 time

daviian edited Jun 7, 2018 (most recent)

daviian created Jun 7, 2018

CountMurphy commented Jun 7, 2018 •

edited

Edited 3 times

CountMurphy edited Jun 7, 2018 (most recent)

CountMurphy edited Jun 7, 2018

CountMurphy edited Jun 7, 2018

CountMurphy created Jun 7, 2018

xcolour commented Jun 7, 2018 •

edited

Edited 1 time

xcolour edited Jun 7, 2018 (most recent)

xcolour created Jun 7, 2018

mikedilger commented Jun 7, 2018 •

edited

Edited 1 time

mikedilger edited Jun 7, 2018 (most recent)

mikedilger created Jun 7, 2018

christianbundy commented Jun 7, 2018 •

edited

Edited 2 times

christianbundy edited Jun 7, 2018 (most recent)

christianbundy edited Jun 7, 2018

christianbundy created Jun 7, 2018

vmp32k commented Jun 7, 2018 •

edited

Edited 1 time

vmp32k edited Jun 7, 2018 (most recent)

vmp32k created Jun 7, 2018

christianbundy commented Jun 7, 2018 •

edited

Edited 1 time

christianbundy edited Jun 7, 2018 (most recent)

christianbundy created Jun 7, 2018

stanier commented Jun 8, 2018 •

edited

Edited 1 time

stanier edited Jun 8, 2018 (most recent)

stanier created Jun 8, 2018

justinclift commented Jun 8, 2018 •

edited

Edited 1 time

justinclift edited Jun 8, 2018 (most recent)

justinclift created Jun 8, 2018