Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Ensure files are read with the proper encoding #112

Open
wants to merge 1 commit into from

2 participants

@joefiorini

This fixed the problems I had with copying binary files through the pipeline.

This was referenced
@ronaldocpontes

How can I review this pull request?

I am not a Ruby specialist... can I pull the changes directly from the gem location and still get updates for new versions? What would be the git commands for this?

The gem location in my environment is:
C:\RailsInstaller\Ruby1.9.3\lib\ruby\gems\1.9.1\bundler\gems\rake-pipeline-65b1e744defa

Tks

@ronaldocpontes

I am still getting corrupted images...

  • copied your changed "file_wrapper.rb" into C:\RailsInstaller\Ruby1.9.3\lib\ruby\gems\1.9.1\bundler\gems\rake-pipeline-65b1e744defa\lib\rake-pipeline
  • deleted the assets and tmp folder
  • ran bundle exec rake build

This is what I am using in my Assetfile:

concatFilter = Rake::Pipeline::ConcatFilter
concatFilter.processes_binary_files
match "static/*/" do
filter concatFilter do |input|
input.sub(/static\//, '')
end
end

Note: I am sure your changes are being picked up because I get an error if I purposely corrupt "file_wrapper.rb"

@joefiorini
@ronaldocpontes

Yes...

I am posting an original and after-corruption png...
I am running on Windows 8

CORRUPTED

iphone4s_black

ORIGINAL

original-iphone4s_black

@ronaldocpontes

If it helps here is the output from debugging the code:

FILE: img/theme/iphone4s_black.png
before read : BINARY
after read : BINARY
contents.encoding : ASCII-8BIT
forcing binary encoding...
contents.encoding : ASCII-8BIT

FILE: app.js
before read : BINARY
after read : BINARY
contents.encoding : ASCII-8BIT
forcing binary encoding...
contents.encoding : ASCII-8BIT

  def read
    print "\n\FILE: ",path
    print "\n before read      : ", encoding
    contents = if "".respond_to?(:encode)
      File.read(fullpath, :encoding => encoding)
    else
      File.read(fullpath)
    end
    print "\n after read        : ", encoding
    print "\n contents.encoding : ", contents.encoding

    # In our unit tests Rubinius returns false when the encoding is BINARY
    # The encoding type check bypasses the problem and is probably acceptable, but isn't ideal
    if encoding != "BINARY" && "".respond_to?(:encode) && !contents.valid_encoding?
      print "\n forcing binary encoding..."
      contents.force_encoding("BINARY")
      if !contents.valid_encoding?
        raise EncodingError, "The file at the path #{fullpath} is not valid #{encoding}. Please save it again as #{encoding}."
      end
    end
    if encoding == "BINARY" 
      print "\n forcing binary encoding..."
      contents.force_encoding("BINARY")
    end

    print "\n contents.encoding : ", contents.encoding
    contents
  end
@ronaldocpontes

Strangely, the write method also seems to be on binary encoding...

FILE: img/theme/iphone4s_black.png
before read : BINARY
after read : BINARY
contents.encoding : ASCII-8BIT
forcing binary encoding...
contents.encoding : ASCII-8BIT
before write : ASCII-8BIT ->/assets/img/theme/iphone4s_black.png

FILE: app.css
before read : BINARY
after read : BINARY
contents.encoding : ASCII-8BIT
forcing binary encoding...
contents.encoding : ASCII-8BIT
before write : ASCII-8BIT ->assets/app.css

  def write(string)
    raise UnopenedFile unless @created_file
    print "\n before write      : ", string.encoding, " ->", @created_file.path
    @created_file.set_encoding string.encoding
    @created_file.write(string)
  end
@ronaldocpontes

Hi Joe,

Just curious if you had a chance to replicate the issue...

Tks

@joefiorini

@ronaldocpontes yes, just tried it and it appears to be working for me. I'm going to try a slightly more detailed test, just in case.

@joefiorini

Just tried it with your exact setup. Ran the image through the same concat filter you posted above and it worked fine for me. My guess is it's a Windows issue. Does anyone else have a Windows setup they can use to try processing the above image on my fork?

@ronaldocpontes

Hi Joe,

I finally could fix the issue on Windows with following set of methods:
check: http://blog.leosoto.com/2008/03/reading-binary-file-on-ruby.html

  def read
    contents = if "".respond_to?(:encode) and encoding=="BINARY"
        contents = open(fullpath, "rb") {|io| io.read }
    else
      File.read(fullpath)
    end
    contents
  end


  def create
    FileUtils.mkdir_p(File.dirname(fullpath))

    @created_file = if "".respond_to?(:encode) and encoding=="BINARY"
      File.open(fullpath, "wb")
    else
      File.open(fullpath, "w")
    end

    if block_given?
      yield @created_file
    end

    @created_file
  ensure
    if block_given?
      @created_file.close
      @created_file = nil
    end
  end

  def write(string)
    raise UnopenedFile unless @created_file
    @created_file.set_encoding string.encoding
    @created_file.write(string)
  end

Could you test on your OS and update the pull request if you are happy with it?

Thanks

@ronaldocpontes

Note that I removed some encoding manipulation, so for a non-binary file:

FILE: vendor/bootstrap.js
encoding variable : UTF-8
contents.encoding : CP850

Write method:
string.encoding : CP850

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
Showing with 5 additions and 1 deletion.
  1. +5 −1 lib/rake-pipeline/file_wrapper.rb
View
6 lib/rake-pipeline/file_wrapper.rb
@@ -110,7 +110,10 @@ def read
# In our unit tests Rubinius returns false when the encoding is BINARY
# The encoding type check bypasses the problem and is probably acceptable, but isn't ideal
if encoding != "BINARY" && "".respond_to?(:encode) && !contents.valid_encoding?
- raise EncodingError, "The file at the path #{fullpath} is not valid #{encoding}. Please save it again as #{encoding}."
+ contents.force_encoding("BINARY")
+ if !contents.valid_encoding?
+ raise EncodingError, "The file at the path #{fullpath} is not valid #{encoding}. Please save it again as #{encoding}."
+ end
end
contents
@@ -168,6 +171,7 @@ def closed?
# @raise [UnopenedFile] if the file is not already opened.
def write(string)
raise UnopenedFile unless @created_file
+ @created_file.set_encoding string.encoding
@created_file.write(string)
end
Something went wrong with that request. Please try again.