ArgumentError invalid byte sequence in UTF-8

[pepestupid]

# lib/grape/validations/validators/allow_blank_validator.rb
# frozen_string_literal: true

module Grape
  module Validations
    module Validators
      class AllowBlankValidator < Base
        def validate_param!(attr_name, params)
          return if (options_key?(:value) ? @option[:value] : @option) || !params.is_a?(Hash)

          value = params[attr_name]
          value = value.strip if value.respond_to?(:strip)

          return if value == false || value.present?

          raise Grape::Exceptions::Validation.new(params: [@scope.full_name(attr_name)], message: message(:blank))
        end
      end
    end
  end
end

[dblock]

Do you have a repro? Write a failing test for this? Probably some data being sent that is not UTF-8 and we are not handling it right.

[ericproulx]

Probably a hidden character. Something like this fails with invalid byte sequence in UTF-8

"abce \xC3".strip

[dblock]

@pepestupid help figure out whether this is a valid scenario and the validator needs to handle it better

[ericproulx]

I think we should use the blank? method instead of dealing with strip. It already handling the encoding part.

[pepestupid]

@dblock I can't repro. I think I need more time to investigate

[ericproulx]

I can reproduce it with the allow_blank and values validators.

context 'bad encoding' do
  let(:app) do
    Class.new(Grape::API) do
      default_format :json

      params do
        requires :name, type: String, allow_blank: false
      end
      get '/bad_encoding'
    end
  end

  it 'does not raise an error' do
    expect { get '/bad_encoding', name: "Hello \x94\xfa\x97\x7b" }.not_to raise_error
  end
end

context 'bad encoding' do
  let(:app) do
    Class.new(Grape::API) do
      default_format :json

      params do
        requires :name, type: String, values: %w(a b)
      end
      get '/bad_encoding'
    end
  end

  it 'does not raise an error' do
    expect { get '/bad_encoding', name: "Hello \x94\xfa\x97\x7b" }.not_to raise_error
  end
end

As suggested here, we should scrub user input.

[ericproulx]

As of today, its still an issue even in Rails and the debate is still on going about who should handle that. In my opnion, if a lib plays with the received data, it should handle the case.

[ericproulx]

@pepestupid you can try adding this middleware in your stack ? At least with it, you can choose how you want to handle the cases through a strategy.

[ericproulx]

@dblock we'll have to question ourselves about what we want to do about it.

[dblock]

@dblock we'll have to question ourselves about what we want to do about it.

If the API supports non-UTF-8 characters then I'd expect the validator not to blow up on it.

[pepestupid]

As of today, its still an issue even in Rails and the debate is still on going about who should handle that. In my opnion, if a lib plays with the received data, it should handle the case.

thank @ericproulx it helps me a lot