New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SafeBuffer#gsub breaks block form match variables $1, $2, $`, $&, and $’ #1555

Closed
tardate opened this Issue Jun 8, 2011 · 34 comments

Comments

Projects
None yet
@tardate
Contributor

tardate commented Jun 8, 2011

commit 53a2c0b introduced a change to ensure gsub returns an unsafe string.

When gsub is used in block form, variables such as $1, $2, $`, $&, and $’ will be set appropriately. Now it seems that SafeBuffer's intervention loses the context and these variables are no longer available in the block.

This is the simplest test I can make to demonstrate the issue (which currently fails on 3.0-stable)...

class SafeBufferTest < ActiveSupport::TestCase
  def setup
    @buffer = ActiveSupport::SafeBuffer.new
  end
  [..]
  test "Should not break gsub block form match variables" do
    @buffer << 'matchme'
    @buffer.gsub(/(matchme)/) { assert_equal 'matchme', $1 }
  end

So far it's know that this breaks escape_javascript since it uses $1 matcher (see #1553 ). I presume there may be others also affected.

Investigating what a good fix would be.. seems awful heavy handed to have to capture and reset all those variables, but on the other hand, its bad form to leave a ruby core method different from it's documented behaviour.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Jun 8, 2011

Contributor

There is not much we can do here. Maybe we should get rid of gsub altogether. /cc @tenderlove

Contributor

josevalim commented Jun 8, 2011

There is not much we can do here. Maybe we should get rid of gsub altogether. /cc @tenderlove

@tardate

This comment has been minimized.

Show comment
Hide comment
@tardate

tardate Jun 8, 2011

Contributor

Yes, you're right. I've investigated further and this one seems to cross the border into ruby internals. Not 100% sure but might even be considered a ruby bug: just the act of chaining up the inheritance hierarchy with super causes "last match" globs ($1, $2 etc) to be lost from the caller's context.

Action? Removing gsub support for SafeBuffer is one, but the ramifications may be quite widespread. Or just document the modified behaviour of gsub with a SafeBuffer?

Contributor

tardate commented Jun 8, 2011

Yes, you're right. I've investigated further and this one seems to cross the border into ruby internals. Not 100% sure but might even be considered a ruby bug: just the act of chaining up the inheritance hierarchy with super causes "last match" globs ($1, $2 etc) to be lost from the caller's context.

Action? Removing gsub support for SafeBuffer is one, but the ramifications may be quite widespread. Or just document the modified behaviour of gsub with a SafeBuffer?

@tenderlove

This comment has been minimized.

Show comment
Hide comment
@tenderlove

tenderlove Jun 8, 2011

Member

IMO docco is the best solution. We can't remove gsub, and even a regular subclass of string that just calls super will break:

irb(main):001:0> class X < String; def gsub(*args); super; end end
=> nil
irb(main):002:0> X.new('hello').gsub(/(l)/) { $1 + 'm' }
NoMethodError: undefined method `+' for nil:NilClass
    from (irb):2:in `block in irb_binding'
    from (irb):1:in `gsub'
    from (irb):1:in `gsub'
    from (irb):2
    from /Users/aaron/.local/bin/irb:12:in `<main>'
irb(main):003:0>
Member

tenderlove commented Jun 8, 2011

IMO docco is the best solution. We can't remove gsub, and even a regular subclass of string that just calls super will break:

irb(main):001:0> class X < String; def gsub(*args); super; end end
=> nil
irb(main):002:0> X.new('hello').gsub(/(l)/) { $1 + 'm' }
NoMethodError: undefined method `+' for nil:NilClass
    from (irb):2:in `block in irb_binding'
    from (irb):1:in `gsub'
    from (irb):1:in `gsub'
    from (irb):2
    from /Users/aaron/.local/bin/irb:12:in `<main>'
irb(main):003:0>
@nex3

This comment has been minimized.

Show comment
Hide comment
@nex3

nex3 Jun 8, 2011

Contributor

A more serious monkeypatch could solve this, although it may have unfortunate performance implications. If you do something like this:

def gsub(rx, *args)
  super {|s| s =~ rx; yield}
end

you should be able to properly set the magic variables before running the block.

Contributor

nex3 commented Jun 8, 2011

A more serious monkeypatch could solve this, although it may have unfortunate performance implications. If you do something like this:

def gsub(rx, *args)
  super {|s| s =~ rx; yield}
end

you should be able to properly set the magic variables before running the block.

@dolzenko

This comment has been minimized.

Show comment
Hide comment
@dolzenko

dolzenko Jun 9, 2011

Contributor

+1 can't use the Rack::Utils.escape on SafeBuffer now

ree-1.8.7-2010.02 > Rack::Utils.escape('>')
 => "%3E" 

ree-1.8.7-2010.02 > Rack::Utils.escape(ActiveSupport::SafeBuffer.new('>'))
NoMethodError: undefined method `bytesize' for nil:NilClass
Contributor

dolzenko commented Jun 9, 2011

+1 can't use the Rack::Utils.escape on SafeBuffer now

ree-1.8.7-2010.02 > Rack::Utils.escape('>')
 => "%3E" 

ree-1.8.7-2010.02 > Rack::Utils.escape(ActiveSupport::SafeBuffer.new('>'))
NoMethodError: undefined method `bytesize' for nil:NilClass
@tardate

This comment has been minimized.

Show comment
Hide comment
@tardate

tardate Jun 10, 2011

Contributor

I've pushed a suggested fix on #1622 .. (update: and just yanked it because it's no good)

Contributor

tardate commented Jun 10, 2011

I've pushed a suggested fix on #1622 .. (update: and just yanked it because it's no good)

@tardate

This comment has been minimized.

Show comment
Hide comment
@tardate

tardate Jun 19, 2011

Contributor

(per @dmathieu) http://www.ruby-forum.com/topic/198458 - best explanation/discussion I've seen of why the magic matching variables break when you attempt to intercept gsub (and why it's nigh impossible to "just fix it")

Contributor

tardate commented Jun 19, 2011

(per @dmathieu) http://www.ruby-forum.com/topic/198458 - best explanation/discussion I've seen of why the magic matching variables break when you attempt to intercept gsub (and why it's nigh impossible to "just fix it")

@janx

This comment has been minimized.

Show comment
Hide comment
@janx

janx Jun 20, 2011

Contributor

rails_autolink is affected https://github.com/tenderlove/rails_autolink/blob/master/lib/rails_autolink/helpers.rb: Line #59 turns text into safebuffer, Line #85 and #117 invoke gsub{} on it.

Contributor

janx commented Jun 20, 2011

rails_autolink is affected https://github.com/tenderlove/rails_autolink/blob/master/lib/rails_autolink/helpers.rb: Line #59 turns text into safebuffer, Line #85 and #117 invoke gsub{} on it.

jake3030 pushed a commit to jake3030/rails that referenced this issue Jun 28, 2011

Fix for Integration::Session follow_redirect! headers['location'] bug…
… with Rack [#1555 state:resolved]

Signed-off-by: Joshua Peek <josh@joshpeek.com>
@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Jul 3, 2011

Contributor

@janx line 59 in rails_autolink does not return a safebuffer. It uses #to_str, which transforms any safebuffer into a string.

ruby-1.9.2-p180 :004 > "test".html_safe.class
 => ActiveSupport::SafeBuffer 
ruby-1.9.2-p180 :006 > "test".html_safe.to_str.class
=> String 

Therefore it should not have this gsub problem.

Contributor

dmathieu commented Jul 3, 2011

@janx line 59 in rails_autolink does not return a safebuffer. It uses #to_str, which transforms any safebuffer into a string.

ruby-1.9.2-p180 :004 > "test".html_safe.class
 => ActiveSupport::SafeBuffer 
ruby-1.9.2-p180 :006 > "test".html_safe.to_str.class
=> String 

Therefore it should not have this gsub problem.

@janx

This comment has been minimized.

Show comment
Hide comment
@janx

janx Jul 4, 2011

Contributor

@dmathieu You're right. I gave wrong link/lines, the problem cause of our app was lib/rails_autolink.rb in commit fe9d2c7bb4e099a70cdb5419f4b2068e7555cfb0 (v1.0.1), so it's already fixed in edge. Thanks.

Contributor

janx commented Jul 4, 2011

@dmathieu You're right. I gave wrong link/lines, the problem cause of our app was lib/rails_autolink.rb in commit fe9d2c7bb4e099a70cdb5419f4b2068e7555cfb0 (v1.0.1), so it's already fixed in edge. Thanks.

@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Jul 4, 2011

Contributor

I think we should perhaps, either :

  • Send a warning for every safebuffer which uses an unsafe method with a block, saying it might not work.
  • Raise an exception on SafeBuffer#gsub when there is a block.

The second solution seems better to me.
@josevalim, @tenderlove, what do you think ?

Contributor

dmathieu commented Jul 4, 2011

I think we should perhaps, either :

  • Send a warning for every safebuffer which uses an unsafe method with a block, saying it might not work.
  • Raise an exception on SafeBuffer#gsub when there is a block.

The second solution seems better to me.
@josevalim, @tenderlove, what do you think ?

@tonycoco

This comment has been minimized.

Show comment
Hide comment
@tonycoco

tonycoco Jul 13, 2011

Contributor

Agreed. We need to have this raise an exception at the very least.

<%= render('layouts/application.html.erb').gsub(/\{\{\s*([a-z0-9_]+)\s*\}\}/i) do |match|
  respond_to?("#{$1}_for_cobranding") ? send("#{$1}_for_cobranding") : match
end %>

Simple code like this, which I use to do different branding in layouts, just does not work anymore.

A simple fix is just to:

<%= render('layouts/application.html.erb').to_str.gsub(/\{\{\s*([a-z0-9_]+)\s*\}\}/i) do |match|
  respond_to?("#{$1}_for_cobranding") ? send("#{$1}_for_cobranding") : match
end.html_safe %>
Contributor

tonycoco commented Jul 13, 2011

Agreed. We need to have this raise an exception at the very least.

<%= render('layouts/application.html.erb').gsub(/\{\{\s*([a-z0-9_]+)\s*\}\}/i) do |match|
  respond_to?("#{$1}_for_cobranding") ? send("#{$1}_for_cobranding") : match
end %>

Simple code like this, which I use to do different branding in layouts, just does not work anymore.

A simple fix is just to:

<%= render('layouts/application.html.erb').to_str.gsub(/\{\{\s*([a-z0-9_]+)\s*\}\}/i) do |match|
  respond_to?("#{$1}_for_cobranding") ? send("#{$1}_for_cobranding") : match
end.html_safe %>
@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Jul 25, 2011

Contributor

I have disabled gsub and sub from safe buffers. See #2248

Contributor

dmathieu commented Jul 25, 2011

I have disabled gsub and sub from safe buffers. See #2248

@sillylogger

This comment has been minimized.

Show comment
Hide comment
@sillylogger

sillylogger Jul 26, 2011

I just want to point out this change breaks lib/mail/body.rb's delivering html email templates as they're encoded with:

def to_lf
    gsub(/\n|\r\n|\r/) { "\n" }
end

That's going to bite a few people when they find out their emails are all &lt;table&gt;

sillylogger commented Jul 26, 2011

I just want to point out this change breaks lib/mail/body.rb's delivering html email templates as they're encoded with:

def to_lf
    gsub(/\n|\r\n|\r/) { "\n" }
end

That's going to bite a few people when they find out their emails are all &lt;table&gt;

@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Jul 26, 2011

Contributor

yes, but there's no way to detect whether the block uses $* vars or not.

Contributor

dmathieu commented Jul 26, 2011

yes, but there's no way to detect whether the block uses $* vars or not.

@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Jul 26, 2011

Contributor

Basically, gsub shouldn't be used on safe buffers because you'll have unexpected behaviors with block.
That's why my patch removes it.

After, as it works as long as you don't use $* vars, it might be enough to not remove the feature.
Then both my PR and this issue can be closed ...
But I think that's for a core committer to decide.
cc @josevalim @spastorino

Contributor

dmathieu commented Jul 26, 2011

Basically, gsub shouldn't be used on safe buffers because you'll have unexpected behaviors with block.
That's why my patch removes it.

After, as it works as long as you don't use $* vars, it might be enough to not remove the feature.
Then both my PR and this issue can be closed ...
But I think that's for a core committer to decide.
cc @josevalim @spastorino

@steveh

This comment has been minimized.

Show comment
Hide comment
@steveh

steveh Aug 9, 2011

This also breaks things like CGI.escapeHTML and HTMLEntities.new.encode, and their decoding equivalents.

I think if you're going to modify behaviour from ruby-core, it should raise an exception rather than just return nil for $1.

steveh commented Aug 9, 2011

This also breaks things like CGI.escapeHTML and HTMLEntities.new.encode, and their decoding equivalents.

I think if you're going to modify behaviour from ruby-core, it should raise an exception rather than just return nil for $1.

@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Aug 10, 2011

Contributor

@steveh : that's what my PR, #2248 does.

Contributor

dmathieu commented Aug 10, 2011

@steveh : that's what my PR, #2248 does.

@steveh

This comment has been minimized.

Show comment
Hide comment
@steveh

steveh Aug 10, 2011

@dmathieu Ah, managed to miss that completely. Thanks, will watch that PR instead.

steveh commented Aug 10, 2011

@dmathieu Ah, managed to miss that completely. Thanks, will watch that PR instead.

@akaspick

This comment has been minimized.

Show comment
Hide comment
@akaspick

akaspick Aug 13, 2011

Contributor

Just spent way too long dealing with this issue myself realizing Rails was borking gsub.

My vote is to raise an exception if a block is used if core doesn't want to remove it from the list of unsafe methods. In the mean time, I have to force standard gsub usage with String.new(my_text).gsub { }

Contributor

akaspick commented Aug 13, 2011

Just spent way too long dealing with this issue myself realizing Rails was borking gsub.

My vote is to raise an exception if a block is used if core doesn't want to remove it from the list of unsafe methods. In the mean time, I have to force standard gsub usage with String.new(my_text).gsub { }

@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Aug 14, 2011

Contributor

@akaspick : it can't be removed from the list of unsafe methods. Because it's not a safe method. It'll lead to even more confusion.
A cleaner way to do it is :

my_text.to_str.gsub {}
Contributor

dmathieu commented Aug 14, 2011

@akaspick : it can't be removed from the list of unsafe methods. Because it's not a safe method. It'll lead to even more confusion.
A cleaner way to do it is :

my_text.to_str.gsub {}
@janxious

This comment has been minimized.

Show comment
Hide comment
@janxious

janxious Aug 29, 2011

The gsub part of this affected me, by way of HTTParty. It was totally unexpected and very difficult to track down.

janxious commented Aug 29, 2011

The gsub part of this affected me, by way of HTTParty. It was totally unexpected and very difficult to track down.

@josevalim josevalim closed this in e9f48cd Sep 8, 2011

@josevalim josevalim closed this in b4a6e2f Sep 8, 2011

@jrochkind

This comment has been minimized.

Show comment
Hide comment
@jrochkind

jrochkind Sep 14, 2011

Contributor

So gsub isnt' allowed on a safe buffer anymore. So if you want to take a string and html_escape it, and THEN call gsub on it, and mark the results as html_safe.... how the heck do you do it?

Since calling html_escape will result in an html_safe string that exhibits this bug, here's one way:

string = String.new(html_escape(  string ))  # html escaped, but not a SafeBuffer
string.gsub(....).html_safe  # obviously it's your responsibility to make sure whatever you are sub'ing in is safe, as any other place you explciitly call #html_safe to make a SafeBuffer. 

Is there any better way to do this? This definitely has resulted in some confusing situations here. This use case -- wanting to take an un-safe string, html-escape everything in it, and then gsub on it... is THAT crazy of a case, is it? The solution here still doesn't get people out of having to do crazy things like above, although at least it raises instead of failing mysteriously, giving you a clue you have to do something crazy.

Contributor

jrochkind commented Sep 14, 2011

So gsub isnt' allowed on a safe buffer anymore. So if you want to take a string and html_escape it, and THEN call gsub on it, and mark the results as html_safe.... how the heck do you do it?

Since calling html_escape will result in an html_safe string that exhibits this bug, here's one way:

string = String.new(html_escape(  string ))  # html escaped, but not a SafeBuffer
string.gsub(....).html_safe  # obviously it's your responsibility to make sure whatever you are sub'ing in is safe, as any other place you explciitly call #html_safe to make a SafeBuffer. 

Is there any better way to do this? This definitely has resulted in some confusing situations here. This use case -- wanting to take an un-safe string, html-escape everything in it, and then gsub on it... is THAT crazy of a case, is it? The solution here still doesn't get people out of having to do crazy things like above, although at least it raises instead of failing mysteriously, giving you a clue you have to do something crazy.

@dmathieu

This comment has been minimized.

Show comment
Hide comment
@dmathieu

dmathieu Sep 14, 2011

Contributor

Yes, there's a much better way to do this :

string = html_escape(string).to_str  # Not a SafeBuffer
string.gsub(....).html_safe  # obviously it's your responsibility to make sure whatever you are sub'ing in is safe, as any other place you explciitly call #html_safe to make a SafeBuffer. 
Contributor

dmathieu commented Sep 14, 2011

Yes, there's a much better way to do this :

string = html_escape(string).to_str  # Not a SafeBuffer
string.gsub(....).html_safe  # obviously it's your responsibility to make sure whatever you are sub'ing in is safe, as any other place you explciitly call #html_safe to make a SafeBuffer. 
@toupeira

This comment has been minimized.

Show comment
Hide comment
@toupeira

toupeira Aug 19, 2013

I also just spent a good deal of my day tracking down this bug.

Issue #1555 was resolved by pull request #2248 which raises an exception for these methods, but this change was reverted on the same day for some reason. I think raising an exception when called with a block would be far more sensible than knowingly letting it run in a broken state?

toupeira commented Aug 19, 2013

I also just spent a good deal of my day tracking down this bug.

Issue #1555 was resolved by pull request #2248 which raises an exception for these methods, but this change was reverted on the same day for some reason. I think raising an exception when called with a block would be far more sensible than knowingly letting it run in a broken state?

@BattleBrisket

This comment has been minimized.

Show comment
Hide comment
@BattleBrisket

BattleBrisket Oct 8, 2014

Just ran into this problem myself, and lucked into finding this thread. Wanted to add one important note for other travelers.

The solution provided by @dmathieu...

my_text.to_str.gsub {}

worked for me, however note that you must use to_str rather than to_s.

SafeBuffer inherits from String and overrides to_sto return self, but does not touch to_str, which is why it works in this case.

BattleBrisket commented Oct 8, 2014

Just ran into this problem myself, and lucked into finding this thread. Wanted to add one important note for other travelers.

The solution provided by @dmathieu...

my_text.to_str.gsub {}

worked for me, however note that you must use to_str rather than to_s.

SafeBuffer inherits from String and overrides to_sto return self, but does not touch to_str, which is why it works in this case.

@trandaison

This comment has been minimized.

Show comment
Hide comment
@trandaison

trandaison Apr 13, 2017

I run into a problem with escape_javascript. I render a confirm modal to prereview the article content before saving. Anh JS doesn't work if it has an ' or " in the article's content.

$(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article) %>");

I've tried to use .html('<%= ... %>') or .html("<%= ... %>") but the other case won't work.
The problem was solved when I use .to_str() after render()

$(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article).to_str %>");

trandaison commented Apr 13, 2017

I run into a problem with escape_javascript. I render a confirm modal to prereview the article content before saving. Anh JS doesn't work if it has an ' or " in the article's content.

$(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article) %>");

I've tried to use .html('<%= ... %>') or .html("<%= ... %>") but the other case won't work.
The problem was solved when I use .to_str() after render()

$(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article).to_str %>");
@vishalzambre

This comment has been minimized.

Show comment
Hide comment
@vishalzambre

vishalzambre Apr 13, 2017

Contributor

@trandaison have you tried once with raw?

Contributor

vishalzambre commented Apr 13, 2017

@trandaison have you tried once with raw?

@trandaison

This comment has been minimized.

Show comment
Hide comment
@trandaison

trandaison Apr 13, 2017

@vishalzambre I've tried .html_safe and raw, it won't work too :D

trandaison commented Apr 13, 2017

@vishalzambre I've tried .html_safe and raw, it won't work too :D

@westonganger

This comment has been minimized.

Show comment
Hide comment
@westonganger

westonganger Apr 28, 2017

@trandaison see JangoSteve/remotipart#89

$(".yield-section").html("<%= j "#{render "admin/articles/modal_confirm", article: @article}"  %>");

westonganger commented Apr 28, 2017

@trandaison see JangoSteve/remotipart#89

$(".yield-section").html("<%= j "#{render "admin/articles/modal_confirm", article: @article}"  %>");
@trandaison

This comment has been minimized.

Show comment
Hide comment
@trandaison

trandaison May 10, 2017

@westonganger
In case I edit an article, it renders a html text at the end of the page instead of showing a preview modal. Adding .html_safe or raw() will do the job.
But if using .html_safe, when I only change the image of article (others are not changed), it won't render properly, JS won't execute.

I have to do something like this, and it works as well but it seems like a little bit complicated 😄

<% if action_name == "create" || params[:article][:image].present? %>
  $(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article).to_str %>");
<% else %>
  $(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article) %>");
<% end %>

trandaison commented May 10, 2017

@westonganger
In case I edit an article, it renders a html text at the end of the page instead of showing a preview modal. Adding .html_safe or raw() will do the job.
But if using .html_safe, when I only change the image of article (others are not changed), it won't render properly, JS won't execute.

I have to do something like this, and it works as well but it seems like a little bit complicated 😄

<% if action_name == "create" || params[:article][:image].present? %>
  $(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article).to_str %>");
<% else %>
  $(".confirmation-modal").html("<%= j render("articles/modal_confirm", article: @article) %>");
<% end %>
@kreintjes

This comment has been minimized.

Show comment
Hide comment
@kreintjes

kreintjes Aug 24, 2017

Contributor

Just ran into this issue again today (using Rails 5.0 and Ruby 2.4.1) and took quite some time before I realized ActiveSupport::SafeBuffer was the cause. I see there were several attempts at fixes, which were uncompleted or reverted. It would however be nice if we could at least show a warning when this is happening to prevent wasting time.

Note: this is also a problem when using Regexp.last_match (as did I).

Contributor

kreintjes commented Aug 24, 2017

Just ran into this issue again today (using Rails 5.0 and Ruby 2.4.1) and took quite some time before I realized ActiveSupport::SafeBuffer was the cause. I see there were several attempts at fixes, which were uncompleted or reverted. It would however be nice if we could at least show a warning when this is happening to prevent wasting time.

Note: this is also a problem when using Regexp.last_match (as did I).

jfly added a commit to thewca/worldcubeassociation.org that referenced this issue Dec 12, 2017

jfly added a commit to thewca/worldcubeassociation.org that referenced this issue Dec 12, 2017

jfly added a commit to thewca/worldcubeassociation.org that referenced this issue Dec 12, 2017

jfly added a commit to thewca/worldcubeassociation.org that referenced this issue Dec 12, 2017

jfly added a commit to thewca/worldcubeassociation.org that referenced this issue Dec 12, 2017

sikachu added a commit to sikachu/email-spec that referenced this issue Dec 30, 2017

Fix compatibility issue with Mail 2.7.0
In Mail 2.7.0 in an actual Rails application, HTML part of the body now
returns an `ActiveSupport::SafeBuffer` object instead of a String
object. This causes a problem as there is a known issue of how
`SafeBuffer#gsub` with a block form is broken in Rails[1], and that
`HTMLEntities#decode` actually performs a `gsub` with a block
internally[2].

Upon further investigation, however, it seems like the root cause of
this issue might not be on Mail gem, but actually on `ERB::Util.h`
returning `ActiveSupport::SafeBuffer` object instead of `String`[3].

This commit changes `default_part_body` method to call `to_s.to_str` on
the message body so that we'll be able to pass a String object, which
always works with `gsub`, to `HTMLEntities#decode`.

Please note that we need to call `to_s.to_str` on the object because
`ActiveSupport::SafeBuffer` actually overrides `to_s` to return itself
and not the underlying `String` object.

I believe this PR should fix issue #202, #204, and #205, and it's better
to fix the issue here than in `HTMLEntities`.

[1]: rails/rails#1555
[2]: https://github.com/threedaymonk/htmlentities/blob/v4.3.3/lib/htmlentities/decoder.rb#L10-L20
[3]: threedaymonk/htmlentities#33 (comment)
@TylerRick

This comment has been minimized.

Show comment
Hide comment
@TylerRick

TylerRick Feb 13, 2018

Contributor

I just ran into this issue again today (using rails 5.1.4) and wasted quite some time debugging before I realized ActiveSupport::SafeBuffer was the cause. It looks like I am not the only one who continues to get bitten by this.

It looks like this issue was closed by #2248 but then that fix was reverted (why?). I would suggest reopening this until a solution is finally merged. (Closing the issue suggests that it was resolved. If it was resolved, what was the resolution??)

Perhaps raise an error when gsub is used on a SafeBuffer, or at least emit a warning?

Contributor

TylerRick commented Feb 13, 2018

I just ran into this issue again today (using rails 5.1.4) and wasted quite some time debugging before I realized ActiveSupport::SafeBuffer was the cause. It looks like I am not the only one who continues to get bitten by this.

It looks like this issue was closed by #2248 but then that fix was reverted (why?). I would suggest reopening this until a solution is finally merged. (Closing the issue suggests that it was resolved. If it was resolved, what was the resolution??)

Perhaps raise an error when gsub is used on a SafeBuffer, or at least emit a warning?

etagwerker added a commit to email-spec/email-spec that referenced this issue Apr 3, 2018

Fix compatibility issue with Mail 2.7.0
In Mail 2.7.0 in an actual Rails application, HTML part of the body now
returns an `ActiveSupport::SafeBuffer` object instead of a String
object. This causes a problem as there is a known issue of how
`SafeBuffer#gsub` with a block form is broken in Rails[1], and that
`HTMLEntities#decode` actually performs a `gsub` with a block
internally[2].

Upon further investigation, however, it seems like the root cause of
this issue might not be on Mail gem, but actually on `ERB::Util.h`
returning `ActiveSupport::SafeBuffer` object instead of `String`[3].

This commit changes `default_part_body` method to call `to_s.to_str` on
the message body so that we'll be able to pass a String object, which
always works with `gsub`, to `HTMLEntities#decode`.

Please note that we need to call `to_s.to_str` on the object because
`ActiveSupport::SafeBuffer` actually overrides `to_s` to return itself
and not the underlying `String` object.

I believe this PR should fix issue #202, #204, and #205, and it's better
to fix the issue here than in `HTMLEntities`.

[1]: rails/rails#1555
[2]: https://github.com/threedaymonk/htmlentities/blob/v4.3.3/lib/htmlentities/decoder.rb#L10-L20
[3]: threedaymonk/htmlentities#33 (comment)
@gsar

This comment has been minimized.

Show comment
Hide comment
@gsar

gsar Jun 13, 2018

@josevalim @tenderlove this issue should not have been closed, as it still exists in rails 5.1.6. I don't know how many hours have been lost by people chasing down the issue over the years. Please reinstate the warnings that were inexplicably and irresponsibly reverted by e05d4ce#diff-f639a79f308e72f54af369291a4d5907

gsar commented Jun 13, 2018

@josevalim @tenderlove this issue should not have been closed, as it still exists in rails 5.1.6. I don't know how many hours have been lost by people chasing down the issue over the years. Please reinstate the warnings that were inexplicably and irresponsibly reverted by e05d4ce#diff-f639a79f308e72f54af369291a4d5907

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment