Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Giteabot account was compromised #4167

Closed
lafriks opened this issue Jun 7, 2018 · 75 comments
Closed

Giteabot account was compromised #4167

lafriks opened this issue Jun 7, 2018 · 75 comments

Comments

@lafriks
Copy link
Member

@lafriks lafriks commented Jun 7, 2018

We are currently investigating suspiscious activity from an account with write access priviledge to go-gitea organization. A binary was added to releases across multiple go-gitea repositories. We cleaned up all releases and drop temporarily access from the account. We will investigate futhermore to understand what really happen to prevent it in the future and be transparent with you trough the process. In the meantime, if you find any suspicious activity please report them under this issue.

UPDATE: No source code or other Gitea infrastructure was affected, including https://dl.gitea.io/ so it's safe to use it to download Gitea binaries.

GitHub account that was hacked is under full control and also have set 2FA so this should not happen in future again.

What was done:

  • Most of go-gitea organization repositories new release&tag was created with name 0 and added install.exe binary (13KB in size) to that release that was malicious (from our analysis contained crypto currency miner). All these releases and binaries was deleted within 2-3 hours from when they were added.
  • Also 1.4.2 release windows Gitea .exe binary on GitHub was replaced by same 13K binary as above. So if Gitea is working, you are safe.
  • Just in case we did retag 1.4.2 to trigger CI to rebuild binaries so sha256 will be different now as it was before retag.

We have contacted GitHub but have not received any answer from them, yet

UPDATE2:
No actual gitea binaries were compromised so all hashes mentioned in comments below are safe.

SHA256 of malicious .exe file:
bfc5a0358b1ad76ffbc1e1f4670bd3240536e2fbac88272cee3003322a15fffe

UPDATE3:
v1.4.2 has been rereleased at about 2018-06-07 20:00:00 UTC+08

@daviian
Copy link
Member

@daviian daviian commented Jun 7, 2018

In the meantime please be careful when downloading our released binaries, especially the windows ones, until further notice. E.g. keep an eye on the size of the binaries, or a double .exe file ending.
We've found out our .exe binaries within release 1.4.2 were altered as well.

We work hard to follow all trails, clean up and get back to daily business as soon as possible.

@Mauladen
Copy link

@Mauladen Mauladen commented Jun 7, 2018

@daviian Maybe then it is necessary to sign releases?

@daviian
Copy link
Member

@daviian daviian commented Jun 7, 2018

@Mauladen Thank you for your input. We definitely will discuss this.

@OmarAssadi
Copy link
Contributor

@OmarAssadi OmarAssadi commented Jun 7, 2018

Ouch! Yeah, signed binaries would be ideal.

@Mauladen
Copy link

@Mauladen Mauladen commented Jun 7, 2018

default
1
2

@daviian
Copy link
Member

@daviian daviian commented Jun 7, 2018

@Mauladen We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones.

Or did you mean something else?

@lafriks
Copy link
Member Author

@lafriks lafriks commented Jun 7, 2018

This is normal. We retaged 1.4.2 to retrigger CI to rebuild binaries as windows binary for that release was removed and there was added new one malicious one

@axifive
Copy link
Member

@axifive axifive commented Jun 7, 2018

@daviian, Maybe @Mauladen meant that 1.4.2 release commit without GPG signature, but 1.4.1 was

@Mauladen
Copy link

@Mauladen Mauladen commented Jun 7, 2018

Still need to edit the list of changes

@sapk
Copy link
Member

@sapk sapk commented Jun 7, 2018

@axifive commit where not gpg signed by user but github that does it automaticaly (with github key) when the merger is the same the PR author. I wouldn't it consider more safer because if github account were compromised I will been also show as "verified". But we could start to sign tag from now (to be discuss). For information we are, working on signing binary since it is more easy to corrumpt thzan binary as the git commit tree.

@hasufell
Copy link

@hasufell hasufell commented Jun 7, 2018

You should upload a tarball created by git-archive (in addition to those that github automatically generates) which you sign with signify. You can get an idea on how that works/looks here:

Libressl uses the same technology.

@CountMurphy
Copy link

@CountMurphy CountMurphy commented Jun 7, 2018

Is there anymore information on this? What was the payload of the binary? Are users going to be informed about this through a blog post or anything? Were any of the linux binaries compromised? I'm rather concerned about what these things may have done to my servers. I'm doubly concerned that I only found out about this browsing the issue page by chance vs an official notice on the project page/blog

@asiekierka
Copy link

@asiekierka asiekierka commented Jun 7, 2018

Is the Gitea 1.4.2 release safe to update to from source code at this point in time?

@hasufell
Copy link

@hasufell hasufell commented Jun 7, 2018

At this point, it's not clear to me whether only binaries were affected. How did you verify this? Are your branches protected?

@xcolour
Copy link

@xcolour xcolour commented Jun 7, 2018

shasums differ on binaries I downloaded on 6/4 and 6/7 though they are the same number of bytes:

$ sha256sum gitea-1.4.2-linux-amd64*
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d  gitea-1.4.2-linux-amd64.1

$ ls --color=tty -l gitea-1.4.2-linux-amd64*
-rwxr-xr-x 1 matt matt 53674800 Jun  4 20:39 gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 matt matt 53674800 Jun  7 08:18 gitea-1.4.2-linux-amd64.1

@bkero
Copy link

@bkero bkero commented Jun 7, 2018

Feel like hosting those somewhere so folks can tear them apart?

@xcolour
Copy link

@xcolour xcolour commented Jun 7, 2018

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA. I'll put them somewhere more permanent if necessary.

@jatherrien
Copy link

@jatherrien jatherrien commented Jun 7, 2018

Question - is it just the binaries on the website that were affected, or were the repositories also affected? I ask because yesterday I heard about Gitea and cloned and built the v1.4 branch.

@4oo4
Copy link

@4oo4 4oo4 commented Jun 7, 2018

Would really like some more info if the source itself is compromised or just the binaries. I run a script to check for updates/build from git nightly and was lucky enough to miss this because of the timing.

Is the current 1.4.2 release verified to be clean? If we know that's tainted we should pull that ASAP and put it somewhere else for analysis, otherwise people will still be downloading that if they don't see this issue. I agree with @CountMurphy, can we add something to the README so it's really visible?

@mikedilger
Copy link

@mikedilger mikedilger commented Jun 7, 2018

Can we get hashes so we can check if our binaries are affected? My gitea 1.4.1 (linux amd64) looks like this:
$ sha256sum gitea
d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 gitea

@cupnoodle
Copy link

@cupnoodle cupnoodle commented Jun 7, 2018

I just downloaded gitea 1.4.1 linux-amd-64 and I got d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 as the sha256 too

@4oo4
Copy link

@4oo4 4oo4 commented Jun 7, 2018

@christianbundy
Copy link

@christianbundy christianbundy commented Jun 7, 2018

To make it overwhelmingly clear: nobody knows anything and the only safe hashes are the ones currently found here: https://github.com/go-gitea/gitea/releases

For Gitea 1.4.2 on amd64, that means:

c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d

Sorry, I misunderstood #4167 (comment) to mean that both binaries were compromised.


I've edited this post to remove any ambiguities about what we actually know.

@shuhaowu
Copy link

@shuhaowu shuhaowu commented Jun 7, 2018

Hold on: according to this comment (#4167 (comment)), there are two shas e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 4th and c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d from June 7th.

Are we saying that both of these are compromised or just one of them? For the record the one on my server is e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 5th.

@axifive
Copy link
Member

@axifive axifive commented Jun 7, 2018

@christianbundy, Please, don't make hasty conclusions, wait for a response from the Team member

@christianbundy
Copy link

@christianbundy christianbundy commented Jun 7, 2018

Please, don't make hasty conclusions, wait for a response from the Team member

Sorry about that, I thought the file hashes would be posted by a team member immediately and didn't realize that the info was coming from someone else who was also in the dark. Is this the correct channel to watch for the latest developments? I've powered off my server and need more information to tell whether I've been infected.

@shuhaowu
Copy link

@shuhaowu shuhaowu commented Jun 7, 2018

I see your edits crossing out the entry I pointed to. HOWEVER, isn't this too early to draw conclusion from? How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

We're still missing a couple pieces of information (likely non-exhaustive):

  • Which binaries were compromised and what are their checksums?
  • When did the bot account get compromised?
  • Is the source repository compromised (this might be easy to check if you have a version of the repo cloned from some while back, then you can maybe check diffs or evidence of force pushes).

@SharkyRawr
Copy link

@SharkyRawr SharkyRawr commented Jun 7, 2018

I got:

# ls -la gitea-1.4.1-linux-amd64
-rwxr-xr-x 1 gitea gitea 53663640 May  3 08:02 gitea-1.4.1-linux-amd64

With sha256sum d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851

and I just downloaded gitea-1.4.2-linux-amd64:

# ls -la gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 root root 53674800 Jun  7 14:18 gitea-1.4.2-linux-amd64

With sha256sum c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d which matches the provided .sha256 file: gitea-1.4.2-linux-amd64: OK which should be safe. (right?)

[...] We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones. [...]


I uploaded gitea-1.4.2-linux-amd64 for analysis here, but it doesn't really tell anything obvious.

Going to watch this issue and keep my gitea installation offline for the time being.

@vbrandl
Copy link

@vbrandl vbrandl commented Jun 8, 2018

Since you will start signing binary releases, could you also consider signing the Docker images pushed to the registry?

@justinclift
Copy link
Contributor

@justinclift justinclift commented Jun 8, 2018

It seems it wasn't just Gitea that got hit by this, but also https://github.com/opencompany/www.opencompany.org/releases

Just emailed their contact people directly, in case they're not looking at their GitHub issues yet.

@justinclift
Copy link
Contributor

@justinclift justinclift commented Jun 8, 2018

@graystevens Should the pastebin staff be contacted to kill that pastebin address, or is it better to fully analyse the malware first so it's properly understood?

@adelowo
Copy link
Member

@adelowo adelowo commented Jun 8, 2018

Thanks everyone.. Would re download and put my server back up

@graystevens
Copy link

@graystevens graystevens commented Jun 8, 2018

@justinclift I’ll report the paste to PasteBin now - I’ve got a copy of the contents so we can recreate the output for the malware should we need to. Good shout

@sapk
Copy link
Member

@sapk sapk commented Jun 8, 2018

To be fair, go is now has reproducible build : https://blog.filippo.io/reproducing-go-binaries-byte-by-byte/
That maybe cgo for sqlite and some build env that make them not reproducible.

@sapk
Copy link
Member

@sapk sapk commented Jun 8, 2018

Just for information, the previous rebuild and un-touched hash list. If you have those hash it means you have the binary before the rebuild we have done to reset the 1.4.2 release and also before the attack. If it is the case, you can stay with them.

156655506d373241fea6b8955acbe6fdc148f06b49b4f6e46d11cadcd00d7316  gitea-1.4.2-darwin-10.6-386
5730c1a471dfc2c055442360d431c950b3a4f266403c5f327c94a68b88e67a10  gitea-1.4.2-darwin-10.6-amd64
8c3cc2127161695dea512176cd4282711e8c57972f333253cc89597dfab002ef  gitea-1.4.2-linux-386
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
bca75d81c52b9aa57fd05b8f7ce0572abae3e1a008d8ab77840ca08c53975867  gitea-1.4.2-linux-arm-5
3aa791afce2e3450461038e09622fe04f34b891c47db641be50b6cc3f772645e  gitea-1.4.2-linux-arm64
fc73d06621fc23a944a2a8ee3b19fbd8edb972b0cdaefa67248eb885aa4d6240  gitea-1.4.2-linux-arm-6
5b76cd9b84846c09ba3627219a6cad99542a53d8eec71a427a433ab82eaabacf  gitea-1.4.2-linux-arm-7
4fafeabfa069066fd624364b47b5729f8f068545d4cd8a3620774a982937ac16  gitea-1.4.2-linux-mips64le
7d9e0afcd315f0efbfb26cab303b3fee3939fa0e081b0d3f4f480f950d0a9870  gitea-1.4.2-linux-mips64
7f2e3c1163823889bec1fdd34b4135149c83d53b6dc86f186821e51e0f839e53  gitea-1.4.2-linux-mipsle
a1b8338113d4fdb0360582ba18f6fce97722c779493e7d7e07594d78c7c3f003  gitea-1.4.2-linux-mips
82ad50932bd927ead2e7da4e4c575d94f88e0105a82eda4cce1df7c9fc5ba0dc  gitea-1.4.2-windows-4.0-386.exe
86d7332894390ccaedd39be891044d829f54d4cd71fa21fce5dbd9bc3abbce44  gitea-1.4.2-windows-4.0-amd64.exe

@jvanraaij
Copy link

@jvanraaij jvanraaij commented Jun 20, 2018

The install.exe cleanup pass seems to have missed quite a few repositories:

Most of these are old and not exactly relevant, but it's probably not a good idea to keep malware around.

@yasuokav
Copy link
Contributor

@yasuokav yasuokav commented Jun 20, 2018

@lunny

go-xorm
go-tango
go-xweb
goftp
gobook
tango
gobuild

@justinclift
Copy link
Contributor

@justinclift justinclift commented Jun 20, 2018

Ugh, thanks. What's the approach you're using to locate these? It seems that whatever approach the GitHub staff have used hasn't really been 100% effective. 😦

@lunny
Copy link
Member

@lunny lunny commented Jun 21, 2018

@lafriks
Copy link
Member Author

@lafriks lafriks commented Jun 25, 2018

As after our investigation nothing else was affected and no more information was given by GitHub I think this issue can be closed. Anyone to write blog post about this?

@graystevens
Copy link

@graystevens graystevens commented Jun 25, 2018

@lafriks I posted this blog post back within a couple of days of this all kicking off - its more of a look at the malware than the issue at hand, but I'm happy to update it if you feel something is worth documenting 👍

@tboerger
Copy link
Member

@tboerger tboerger commented Jun 25, 2018

I think he wants to write a post-mortem blog post on the gitea blog :)

@rugk
Copy link
Contributor

@rugk rugk commented Jun 25, 2018

@graystevens BTW your site notice link is a 404 one. 😄

@justinclift
Copy link
Contributor

@justinclift justinclift commented Jun 25, 2018

As after our investigation nothing else was affected and no more information was given by GitHub ...

As a data point, although GitHub haven't said anything about this in public, privately they've responded (via email) to say effectively "Thanks, we're investigating."

The above comments by @jvanraaij and @yasuokav seemed to help too, as (weirdly, from my PoV) GitHub don't seem to have found those particular instances of the malware prior to that.

@justinclift
Copy link
Contributor

@justinclift justinclift commented Jun 25, 2018

For example, all of the repo's listed by @yasuokav here still have the malware: crossoverJie/SSM#36

I'll email GitHub staff again to point it out. They seem to get things taken care of within a few days when contacted directly with an exact list to look at.

@tboerger
Copy link
Member

@tboerger tboerger commented Jun 25, 2018

This is sad for all the other projects, but at least for Gitea we have solved the issues properly, hopefully... :)

@techknowlogick
Copy link
Member

@techknowlogick techknowlogick commented Jul 21, 2018

Closing this issue as binaries are now signed, and 2FA is implemented.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Linked pull requests

Successfully merging a pull request may close this issue.

None yet