Trilogy adapter incompatible with non-ascii column names #459

D-system · 2024-01-04T10:33:50Z

Hi,

Rails added a new official mysql adapter called Trilogy which is pure Ruby (no native mysql lib needed). Trilogy is inside Rails 7.1. For previous versions of Rails, there's a gem https://rubygems.org/gems/activerecord-trilogy-adapter.
The mysql2 adapter does not have that issue.

Blazer: github master branch (ccad905)
Rails: 6.1.7.6
Ruby 3.2.2
Database: mysql with Trilogy adapter
Error: ActionView::Template::Error (incompatible character encodings: UTF-8 and US-ASCII)

The issue is with non-ascii (Japanese characters in UTF-8 in that case).
Strangely enough is does work with a single column retrieve. Multiple columns with 1 column or all columns in UTF-8 does not work.

select a.id as "あ" from jans a LIMIT 2; -- Works: 1 UTF-8
select a.jan_code, a.id from jans a LIMIT 2; -- Works: 2 ascii


select a.jan_code, a.id as "あ" from jans a LIMIT 2; -- Fails: 1 UTF-8 and 1 ascii
select a.jan_code as "え", a.id as "あ" from jans a LIMIT 2; -- Fails: 2 UTF-8

Looking at the code, a dirty fix would be to change

blazer/app/controllers/blazer/queries_controller.rb

Line 156 in ccad905

@columns = @result.columns

with:

# The `dup` is for the frozen string
@columns = @result.columns.map { _1.dup.force_encoding("utf-8") }

Line 118 probably need that change too.

Thanks!

Bracetrace

Started POST "/blazer/queries/run" for ::1 at 2024-01-04 19:09:44 +0900
EcmsUser Load (0.6ms) SELECT ecms_users.* FROM ecms_users WHERE ecms_users.deleted_at IS NULL AND ecms_users.id = 1 ORDER BY ecms_users.id ASC LIMIT 1
Processing by Blazer::QueriesController#run as HTML
Parameters: {"statement"=>"select a.jan_code, a.id as "あ"\r\nfrom jans a\r\nLIMIT 2\r\n;\r\n\r\n", "data_source"=>"main", "run_id"=>"ec57aeab-29a4-4aff-8a41-e83ab0a4810c"}
TRANSACTION (0.3ms) BEGIN
Blazer::Audit Create (1.9ms) INSERT INTO blazer_audits (user_id, statement, data_source, created_at) VALUES (1, 'select a.jan_code, a.id as "あ"\r\nfrom jans a\r\nLIMIT 2\r\n;\r\n\r\n', 'main', '2024-01-04 19:09:44')
TRANSACTION (0.5ms) COMMIT
TRANSACTION (0.2ms) BEGIN
(1.3ms) SET max_execution_time = 30000
(0.3ms) select a.jan_code, a.id as "あ"
from jans a
LIMIT 2
;

/blazer,user_id:1,user_name: Nakumura,run_id:ec57aeab-29a4-4aff-8a41-e83ab0a4810c/
TRANSACTION (0.2ms) ROLLBACK
Rendering /Users/thomas/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/blazer-ccad90553237/app/views/blazer/queries/run.html.erb
Rendered /Users/thomas/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/blazer-ccad90553237/app/views/blazer/queries/_caching.html.erb (Duration: 0.0ms | Allocations: 9)
hash_method_missing to_ary /Users/thomas/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/chartkick-5.0.5/lib/chartkick/helper.rb:91:in %' hash_method_missing to_ary /Users/thomas/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/chartkick-5.0.5/lib/chartkick/helper.rb:103:in %'
Rendered /Users/thomas/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/bundler/gems/blazer-ccad90553237/app/views/blazer/queries/run.html.erb (Duration: 0.7ms | Allocations: 748)
Completed 500 Internal Server Error in 15ms (ActiveRecord: 4.7ms | SQL Queries: 7 (0 cached) | Allocations: 8685)

ActionView::Template::Error (incompatible character encodings: UTF-8 and US-ASCII):
139: <% type = @column_types[i] %>
140:
141:

142: <%= key %>
143:

144:
145: <% end %>

activesupport (6.1.7.6) lib/active_support/core_ext/string/output_safety.rb:215:in concat' activesupport (6.1.7.6) lib/active_support/core_ext/string/output_safety.rb:215:in concat'
actionview (6.1.7.6) lib/action_view/buffers.rb:29:in <<' blazer (ccad90553237) app/views/blazer/queries/run.html.erb:142 blazer (ccad90553237) app/views/blazer/queries/run.html.erb:138:in each'
blazer (ccad905) app/views/blazer/queries/run.html.erb:138:in each_with_index' blazer (ccad90553237) app/views/blazer/queries/run.html.erb:138 actionview (6.1.7.6) lib/action_view/base.rb:247:in public_send'
actionview (6.1.7.6) lib/action_view/base.rb:247:in _run' actionview (6.1.7.6) lib/action_view/template.rb:154:in block in render'
activesupport (6.1.7.6) lib/active_support/notifications.rb:203:in block in instrument' activesupport (6.1.7.6) lib/active_support/notifications/instrumenter.rb:24:in instrument'
activesupport (6.1.7.6) lib/active_support/notifications.rb:203:in instrument' actionview (6.1.7.6) lib/action_view/template.rb:345:in instrument_render_template'
actionview (6.1.7.6) lib/action_view/template.rb:152:in render' actionview (6.1.7.6) lib/action_view/renderer/template_renderer.rb:61:in block (2 levels) in render_template'
activesupport (6.1.7.6) lib/active_support/notifications.rb:203:in block in instrument' activesupport (6.1.7.6) lib/active_support/notifications/instrumenter.rb:24:in instrument'
activesupport (6.1.7.6) lib/active_support/notifications.rb:203:in instrument' actionview (6.1.7.6) lib/action_view/renderer/template_renderer.rb:56:in block in render_template'
actionview (6.1.7.6) lib/action_view/renderer/template_renderer.rb:75:in render_with_layout' actionview (6.1.7.6) lib/action_view/renderer/template_renderer.rb:55:in render_template'
actionview (6.1.7.6) lib/action_view/renderer/template_renderer.rb:11:in render' actionview (6.1.7.6) lib/action_view/renderer/renderer.rb:61:in render_template_to_object'
actionview (6.1.7.6) lib/action_view/renderer/renderer.rb:29:in render_to_object' actionview (6.1.7.6) lib/action_view/rendering.rb:117:in block in _render_template'
actionview (6.1.7.6) lib/action_view/base.rb:273:in in_rendering_context' actionview (6.1.7.6) lib/action_view/rendering.rb:116:in _render_template'
actionpack (6.1.7.6) lib/action_controller/metal/streaming.rb:218:in _render_template' actionview (6.1.7.6) lib/action_view/rendering.rb:103:in render_to_body'
actionpack (6.1.7.6) lib/action_controller/metal/rendering.rb:52:in render_to_body' actionpack (6.1.7.6) lib/action_controller/metal/renderers.rb:142:in render_to_body'
actionpack (6.1.7.6) lib/abstract_controller/rendering.rb:25:in render' actionpack (6.1.7.6) lib/action_controller/metal/rendering.rb:36:in render'
actionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:46:in block (2 levels) in render' benchmark (0.3.0) lib/benchmark.rb:313:in realtime'
activesupport (6.1.7.6) lib/active_support/core_ext/benchmark.rb:14:in ms' actionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:46:in block in render'
actionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:86:in cleanup_view_runtime' activerecord (6.1.7.6) lib/active_record/railties/controller_runtime.rb:34:in cleanup_view_runtime'
actionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:45:in render' blazer (ccad90553237) app/controllers/blazer/queries_controller.rb:268:in block (2 levels) in render_run'
actionpack (6.1.7.6) lib/action_controller/metal/mime_responds.rb:214:in respond_to' blazer (ccad90553237) app/controllers/blazer/queries_controller.rb:266:in render_run'
blazer (ccad905) app/controllers/blazer/queries_controller.rb:169:in run' actionpack (6.1.7.6) lib/action_controller/metal/basic_implicit_render.rb:6:in send_action'
actionpack (6.1.7.6) lib/abstract_controller/base.rb:228:in process_action' actionpack (6.1.7.6) lib/action_controller/metal/rendering.rb:30:in process_action'
actionpack (6.1.7.6) lib/abstract_controller/callbacks.rb:42:in block in process_action' activesupport (6.1.7.6) lib/active_support/callbacks.rb:106:in run_callbacks'
actionpack (6.1.7.6) lib/abstract_controller/callbacks.rb:41:in process_action' actionpack (6.1.7.6) lib/action_controller/metal/rescue.rb:22:in process_action'
actionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:34:in block in process_action' activesupport (6.1.7.6) lib/active_support/notifications.rb:203:in block in instrument'
activesupport (6.1.7.6) lib/active_support/notifications/instrumenter.rb:24:in instrument' activesupport (6.1.7.6) lib/active_support/notifications.rb:203:in instrument'
actionpack (6.1.7.6) lib/action_controller/metal/instrumentation.rb:33:in process_action' actionpack (6.1.7.6) lib/action_controller/metal/params_wrapper.rb:249:in process_action'
activerecord (6.1.7.6) lib/active_record/railties/controller_runtime.rb:27:in process_action' query_count (1.1.1) lib/query_count/controller_runtime.rb:26:in process_action'
actionpack (6.1.7.6) lib/abstract_controller/base.rb:165:in process' actionview (6.1.7.6) lib/action_view/rendering.rb:39:in process'
actionpack (6.1.7.6) lib/action_controller/metal.rb:190:in dispatch' actionpack (6.1.7.6) lib/action_controller/metal.rb:254:in dispatch'
actionpack (6.1.7.6) lib/action_dispatch/routing/route_set.rb:50:in dispatch' actionpack (6.1.7.6) lib/action_dispatch/routing/route_set.rb:33:in serve'
actionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:50:in block in serve' actionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:32:in each'
actionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:32:in serve' actionpack (6.1.7.6) lib/action_dispatch/routing/route_set.rb:842:in call'
railties (6.1.7.6) lib/rails/engine.rb:539:in call' railties (6.1.7.6) lib/rails/railtie.rb:207:in public_send'
railties (6.1.7.6) lib/rails/railtie.rb:207:in method_missing' actionpack (6.1.7.6) lib/action_dispatch/routing/mapper.rb:20:in block in class:Constraints'
actionpack (6.1.7.6) lib/action_dispatch/routing/mapper.rb:49:in serve' actionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:50:in block in serve'
actionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:32:in each' actionpack (6.1.7.6) lib/action_dispatch/journey/router.rb:32:in serve'
actionpack (6.1.7.6) lib/action_dispatch/routing/route_set.rb:842:in call' coverband (6.0.1) lib/coverband/integrations/background_middleware.rb:10:in call'
exception_notification (4.5.0) lib/exception_notification/rack.rb:49:in call' warden (1.2.9) lib/warden/manager.rb:36:in block in call'
warden (1.2.9) lib/warden/manager.rb:34:in catch' warden (1.2.9) lib/warden/manager.rb:34:in call'
rack (2.2.8) lib/rack/tempfile_reaper.rb:15:in call' rack (2.2.8) lib/rack/etag.rb:27:in call'
rack (2.2.8) lib/rack/conditional_get.rb:40:in call' rack (2.2.8) lib/rack/head.rb:12:in call'
actionpack (6.1.7.6) lib/action_dispatch/http/permissions_policy.rb:22:in call' actionpack (6.1.7.6) lib/action_dispatch/http/content_security_policy.rb:19:in call'
rack (2.2.8) lib/rack/session/abstract/id.rb:266:in context' rack (2.2.8) lib/rack/session/abstract/id.rb:260:in call'
actionpack (6.1.7.6) lib/action_dispatch/middleware/cookies.rb:697:in call' activerecord (6.1.7.6) lib/active_record/migration.rb:601:in call'
actionpack (6.1.7.6) lib/action_dispatch/middleware/callbacks.rb:27:in block in call' activesupport (6.1.7.6) lib/active_support/callbacks.rb:98:in run_callbacks'
actionpack (6.1.7.6) lib/action_dispatch/middleware/callbacks.rb:26:in call' actionpack (6.1.7.6) lib/action_dispatch/middleware/executor.rb:14:in call'
actionpack (6.1.7.6) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in call' actionpack (6.1.7.6) lib/action_dispatch/middleware/debug_exceptions.rb:29:in call'
actionpack (6.1.7.6) lib/action_dispatch/middleware/show_exceptions.rb:33:in call' railties (6.1.7.6) lib/rails/rack/logger.rb:37:in call_app'
railties (6.1.7.6) lib/rails/rack/logger.rb:26:in block in call' activesupport (6.1.7.6) lib/active_support/tagged_logging.rb:99:in block in tagged'
activesupport (6.1.7.6) lib/active_support/tagged_logging.rb:37:in tagged' activesupport (6.1.7.6) lib/active_support/tagged_logging.rb:99:in tagged'
railties (6.1.7.6) lib/rails/rack/logger.rb:26:in call' sprockets-rails (3.4.2) lib/sprockets/rails/quiet_assets.rb:13:in call'
actionpack (6.1.7.6) lib/action_dispatch/middleware/remote_ip.rb:81:in call' request_store (1.5.1) lib/request_store/middleware.rb:19:in call'
actionpack (6.1.7.6) lib/action_dispatch/middleware/request_id.rb:26:in call' rack (2.2.8) lib/rack/method_override.rb:24:in call'
rack (2.2.8) lib/rack/runtime.rb:22:in call' activesupport (6.1.7.6) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call'
actionpack (6.1.7.6) lib/action_dispatch/middleware/executor.rb:14:in call' actionpack (6.1.7.6) lib/action_dispatch/middleware/static.rb:24:in call'
rack (2.2.8) lib/rack/sendfile.rb:110:in call' actionpack (6.1.7.6) lib/action_dispatch/middleware/host_authorization.rb:148:in call'
rack-cors (2.0.1) lib/rack/cors.rb:102:in call' railties (6.1.7.6) lib/rails/engine.rb:539:in call'
puma (6.4.0) lib/puma/configuration.rb:272:in call' puma (6.4.0) lib/puma/request.rb:100:in block in handle_request'
puma (6.4.0) lib/puma/thread_pool.rb:378:in with_force_shutdown' puma (6.4.0) lib/puma/request.rb:99:in handle_request'
puma (6.4.0) lib/puma/server.rb:443:in process_client' puma (6.4.0) lib/puma/server.rb:241:in block in run'
puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'

The text was updated successfully, but these errors were encountered:

D-system · 2024-01-05T00:40:04Z

Another solution less dirty that can lives as a monkey patch for now and unblock the situation:

# config/initializers/blazer_utf8.rb
module Blazer
  class Result
    def columns
      @columns.map { _1.dup.force_encoding("utf-8") }
    end
  end
end

ankane · 2024-01-06T21:43:13Z

Hi @D-system, thanks for reporting. It looks like it only causes an error when the query generates a chart:

SELECT 'a' AS "え", 1 AS "あ";   /* error */
SELECT 'a' AS "え", 'b' AS "あ"; /* no error */

but haven't identified the cause.

ankane · 2024-01-06T22:21:04Z

It looks like that's just the behavior of Action View, as this generates the same error:

<%= "え" %>
<%= "え".force_encoding(Encoding::US_ASCII) %>

Fixed in the commit above. Thanks again!

D-system · 2024-01-09T00:21:56Z

Thanks @ankane for the rapid fix 🙇‍♂️

ankane closed this as completed in fd04dd0 Jan 6, 2024

tiagotex mentioned this issue Mar 18, 2024

Incompatible character encodings error while rendering results #468

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trilogy adapter incompatible with non-ascii column names #459

Trilogy adapter incompatible with non-ascii column names #459

D-system commented Jan 4, 2024

D-system commented Jan 5, 2024

ankane commented Jan 6, 2024

ankane commented Jan 6, 2024

D-system commented Jan 9, 2024

Trilogy adapter incompatible with non-ascii column names #459

Trilogy adapter incompatible with non-ascii column names #459

Comments

D-system commented Jan 4, 2024

D-system commented Jan 5, 2024

ankane commented Jan 6, 2024

ankane commented Jan 6, 2024

D-system commented Jan 9, 2024