Developer strategy, at least, have problems with UTF-8 characters #550

Closed
pupeno opened this Issue Dec 7, 2011 · 4 comments

Comments

Projects
None yet
2 participants
@pupeno
Contributor

pupeno commented Dec 7, 2011

Hello,

I'm not totally sure where the problem is, but if I enter my name, "J. Pablo Fernández" and try to store it in the database, I get this error: "invalid byte sequence in UTF-8"

The info hash, when viewed as UTF-8 looks like this: #<OmniAuth::AuthHash::InfoHash email="pupeno@pupeno.com" name="J. Pablo Fern\xE1ndez">. I think the browser is sending the information as UTF-8 and then it's being broken somewhere else.

This would be quite concerning if it affects other strategies.

Cheers!

@mbleigh

This comment has been minimized.

Show comment Hide comment
@mbleigh

mbleigh Feb 8, 2012

Contributor

Could you paste some code that can reproduce this issue? I'd like to either address it or close it.

Contributor

mbleigh commented Feb 8, 2012

Could you paste some code that can reproduce this issue? I'd like to either address it or close it.

@pupeno

This comment has been minimized.

Show comment Hide comment
@pupeno

pupeno Feb 8, 2012

Contributor

Well, on the omniauth.rb initializer I have:

Rails.application.config.middleware.use OmniAuth::Builder do
  provider :developer if Rails.env.development?
  #....
end

then I go to /auth/developer and type J. Pablo Fernández as name, and pupeno@pupeno.com as Email, like this: http://i.imgur.com/65JYC.png and then I get the exception:

PGError: ERROR:  invalid byte sequence for encoding "UTF8": 0xe16e64

The parameters look like this:

{"name"=>"J. Pablo Fern\xE1ndez",
 "email"=>"pupeno+2@pupeno.com",
 "provider"=>"developer"}

instead of

{"name"=>"J. Pablo Fernández",
 "email"=>"pupeno+2@pupeno.com",
 "provider"=>"developer"}

The trace up to where my app starts:

/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:1034:in `get_last_result'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:1034:in `exec_cache'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block in exec_query'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:244:in `block in log'
/gems/activesupport-3.1.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:239:in `log'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/instrumentation/active_record.rb:31:in `block in log_with_newrelic_instrumentation'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/instrumentation/active_record.rb:28:in `log_with_newrelic_instrumentation'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_query'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:63:in `exec_insert'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:90:in `insert'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `insert'
/gems/activerecord-3.1.3/lib/active_record/relation.rb:70:in `insert'
/gems/activerecord-3.1.3/lib/active_record/persistence.rb:313:in `create'
/gems/activerecord-3.1.3/lib/active_record/timestamp.rb:51:in `create'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:268:in `block in create'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:390:in `_run_create_callbacks'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `run_callbacks'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:268:in `create'
/gems/activerecord-3.1.3/lib/active_record/persistence.rb:294:in `create_or_update'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:438:in `block in _run_save_callbacks'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:207:in `block in _conditional_callback_around_90545'
/gems/state_machine-1.1.2/lib/state_machine/integrations/active_record.rb:480:in `block in around_save'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:149:in `block in run_actions'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:169:in `catch_exceptions'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:147:in `run_actions'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:132:in `run_callbacks'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:211:in `run_callbacks'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `block (2 levels) in perform'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `catch'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `block in perform'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:185:in `within_transaction'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:62:in `perform'
/gems/state_machine-1.1.2/lib/state_machine/integrations/active_record.rb:480:in `around_save'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:302:in `_callback_around_90543'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:206:in `_conditional_callback_around_90545'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:390:in `_run_save_callbacks'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `run_callbacks'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:264:in `create_or_update'
/gems/activerecord-3.1.3/lib/active_record/persistence.rb:57:in `save!'
/gems/activerecord-3.1.3/lib/active_record/validations.rb:56:in `save!'
/gems/activerecord-3.1.3/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:246:in `block in save!'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:208:in `transaction'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:491:in `block in transaction_with_trace_ActiveRecord_self_name_transaction'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:486:in `transaction_with_trace_ActiveRecord_self_name_transaction'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:246:in `save!'
app/controllers/tenant/sessions_controller.rb:70:in `create_from_omniauth'

and the create_from_omniauth method contains, essentially:

 def create_from_omniauth
  auth_data = request.env["omniauth.auth"]
  auth = Authentication.find_or_initialize_by_provider_and_uid(auth_data["provider"], auth_data["uid"])

  auth.info = auth_data["info"]
  auth.credentials = auth_data["credentials"]
  auth.extra = auth_data["extra"]

  if auth.user.nil?
    user = User.new
    user.first_name = auth_data["info"]["first_name"] || auth_data["info"]["name"]
    user.last_name = auth_data["info"]["last_name"]
    user.phone_number = auth_data["info"]["phone"]
    user.save!
    auth.user = user
    auth.save!     
  end
end

The user.save! line is the one failing. I could obviously massage the data, but since this is happening on the developer strategy and it seems to never have happened on the while, my bet is on it being an issue in the strategy and not omniauth itself and I don't want to mess with correct twitter and facebook handling of utf-8 data.

Contributor

pupeno commented Feb 8, 2012

Well, on the omniauth.rb initializer I have:

Rails.application.config.middleware.use OmniAuth::Builder do
  provider :developer if Rails.env.development?
  #....
end

then I go to /auth/developer and type J. Pablo Fernández as name, and pupeno@pupeno.com as Email, like this: http://i.imgur.com/65JYC.png and then I get the exception:

PGError: ERROR:  invalid byte sequence for encoding "UTF8": 0xe16e64

The parameters look like this:

{"name"=>"J. Pablo Fern\xE1ndez",
 "email"=>"pupeno+2@pupeno.com",
 "provider"=>"developer"}

instead of

{"name"=>"J. Pablo Fernández",
 "email"=>"pupeno+2@pupeno.com",
 "provider"=>"developer"}

The trace up to where my app starts:

/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:1034:in `get_last_result'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:1034:in `exec_cache'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block in exec_query'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:244:in `block in log'
/gems/activesupport-3.1.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:239:in `log'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/instrumentation/active_record.rb:31:in `block in log_with_newrelic_instrumentation'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/instrumentation/active_record.rb:28:in `log_with_newrelic_instrumentation'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_query'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:63:in `exec_insert'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:90:in `insert'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `insert'
/gems/activerecord-3.1.3/lib/active_record/relation.rb:70:in `insert'
/gems/activerecord-3.1.3/lib/active_record/persistence.rb:313:in `create'
/gems/activerecord-3.1.3/lib/active_record/timestamp.rb:51:in `create'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:268:in `block in create'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:390:in `_run_create_callbacks'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `run_callbacks'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:268:in `create'
/gems/activerecord-3.1.3/lib/active_record/persistence.rb:294:in `create_or_update'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:438:in `block in _run_save_callbacks'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:207:in `block in _conditional_callback_around_90545'
/gems/state_machine-1.1.2/lib/state_machine/integrations/active_record.rb:480:in `block in around_save'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:149:in `block in run_actions'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:169:in `catch_exceptions'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:147:in `run_actions'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:132:in `run_callbacks'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:211:in `run_callbacks'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `block (2 levels) in perform'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `catch'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:63:in `block in perform'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:185:in `within_transaction'
/gems/state_machine-1.1.2/lib/state_machine/transition_collection.rb:62:in `perform'
/gems/state_machine-1.1.2/lib/state_machine/integrations/active_record.rb:480:in `around_save'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:302:in `_callback_around_90543'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:206:in `_conditional_callback_around_90545'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:390:in `_run_save_callbacks'
/gems/activesupport-3.1.3/lib/active_support/callbacks.rb:81:in `run_callbacks'
/gems/activerecord-3.1.3/lib/active_record/callbacks.rb:264:in `create_or_update'
/gems/activerecord-3.1.3/lib/active_record/persistence.rb:57:in `save!'
/gems/activerecord-3.1.3/lib/active_record/validations.rb:56:in `save!'
/gems/activerecord-3.1.3/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:246:in `block in save!'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:208:in `transaction'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:491:in `block in transaction_with_trace_ActiveRecord_self_name_transaction'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped'
/gems/newrelic_rpm-3.3.1/lib/new_relic/agent/method_tracer.rb:486:in `transaction_with_trace_ActiveRecord_self_name_transaction'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/gems/activerecord-3.1.3/lib/active_record/transactions.rb:246:in `save!'
app/controllers/tenant/sessions_controller.rb:70:in `create_from_omniauth'

and the create_from_omniauth method contains, essentially:

 def create_from_omniauth
  auth_data = request.env["omniauth.auth"]
  auth = Authentication.find_or_initialize_by_provider_and_uid(auth_data["provider"], auth_data["uid"])

  auth.info = auth_data["info"]
  auth.credentials = auth_data["credentials"]
  auth.extra = auth_data["extra"]

  if auth.user.nil?
    user = User.new
    user.first_name = auth_data["info"]["first_name"] || auth_data["info"]["name"]
    user.last_name = auth_data["info"]["last_name"]
    user.phone_number = auth_data["info"]["phone"]
    user.save!
    auth.user = user
    auth.save!     
  end
end

The user.save! line is the one failing. I could obviously massage the data, but since this is happening on the developer strategy and it seems to never have happened on the while, my bet is on it being an issue in the strategy and not omniauth itself and I don't want to mess with correct twitter and facebook handling of utf-8 data.

@mbleigh

This comment has been minimized.

Show comment Hide comment
@mbleigh

mbleigh Mar 21, 2012

Contributor

Is this still a problem? Anyone have any ideas what's happening here?

Contributor

mbleigh commented Mar 21, 2012

Is this still a problem? Anyone have any ideas what's happening here?

@pupeno

This comment has been minimized.

Show comment Hide comment
@pupeno

pupeno Mar 21, 2012

Contributor

Yes, I'm still getting the error. I think it's Firefox picking the wrong encoding to send data in. If I force Firefox to use UTF-8 then it works properly. Looking at the sources of the page that prompts me for name and email I see it doesn't specify an encoding, so adding this might actually fix this issue:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Contributor

pupeno commented Mar 21, 2012

Yes, I'm still getting the error. I think it's Firefox picking the wrong encoding to send data in. If I force Firefox to use UTF-8 then it works properly. Looking at the sources of the page that prompts me for name and email I see it doesn't specify an encoding, so adding this might actually fix this issue:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

@mbleigh mbleigh closed this in 8e48c7b Apr 12, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment