Skip to content
This repository has been archived by the owner on Jul 13, 2023. It is now read-only.
This repository has been archived by the owner on Jul 13, 2023. It is now read-only.

Custom processor bug - only on downloaded files #2586

Closed
MadeInSyria opened this issue Apr 6, 2018 · 1 comment
Closed

Custom processor bug - only on downloaded files #2586

MadeInSyria opened this issue Apr 6, 2018 · 1 comment

Comments

@MadeInSyria
Copy link

MadeInSyria commented Apr 6, 2018 •
edited
Loading

Ruby : 2.3.0
Rails : 5.1.4
Paperclip : 6.0.0
AWS-SDK : 3.0.1

I have a bit of a special custom processor. It uses a conversion micro service written in house. Things get funky when I try to return the converted file after downloading it. Below is my non functional processor.

module Paperclip
  class Convertion < Processor

    def initialize(file, options = {}, attachment = nil)
      super
      @file = file
      @format = options[:format]
      @basename = File.basename(@file.path, File.extname(@file.path))
    end

    # Main method for the processor, reference :
    # https://github.com/thoughtbot/paperclip#custom-attachment-processors
    #
    # @return [File] The processed file
    def make
      converted = RestClient::Request.execute(
       method: :post,
       url: 'http://localhost:5678/foobar',
       payload: {
         format: @format,
         file: @file
       },
       raw_response: true)

      return_file = Tempfile.new([@basename, @target_format])
      return_file.binmode
      converted.file.open
      return_file.write(converted.file.read)
      return_file
    end
  end
end

On upload, this processor fails with a Closed Stream error, caused by a file.rewind.

IOError (closed stream):

/Users/foobar/.rbenv/versions/2.3.0/lib/ruby/2.3.0/delegate.rb:341:in `rewind'
/Users/foobar/.rbenv/versions/2.3.0/lib/ruby/2.3.0/delegate.rb:341:in `block in delegating_block'
paperclip (6.0.0) lib/paperclip/io_adapters/abstract_adapter.rb:8:in `rewind'
paperclip (6.0.0) lib/paperclip/storage/s3.rb:392:in `block in flush_writes'
paperclip (6.0.0) lib/paperclip/storage/s3.rb:352:in `each'
paperclip (6.0.0) lib/paperclip/storage/s3.rb:352:in `flush_writes'
paperclip (6.0.0) lib/paperclip/attachment.rb:247:in `save'
paperclip (6.0.0) lib/paperclip/has_attached_file.rb:92:in `block in add_active_record_callbacks'
activesupport (5.1.4) lib/active_support/callbacks.rb:413:in `instance_exec'
activesupport (5.1.4) lib/active_support/callbacks.rb:413:in `block in make_lambda'
activesupport (5.1.4) lib/active_support/callbacks.rb:235:in `block in halting_and_conditional'
activesupport (5.1.4) lib/active_support/callbacks.rb:511:in `block in invoke_after'
activesupport (5.1.4) lib/active_support/callbacks.rb:511:in `each'
activesupport (5.1.4) lib/active_support/callbacks.rb:511:in `invoke_after'
activesupport (5.1.4) lib/active_support/callbacks.rb:132:in `run_callbacks'
activesupport (5.1.4) lib/active_support/callbacks.rb:827:in `_run_save_callbacks'
activerecord (5.1.4) lib/active_record/callbacks.rb:336:in `create_or_update'
activerecord (5.1.4) lib/active_record/persistence.rb:129:in `save'
activerecord (5.1.4) lib/active_record/validations.rb:44:in `save'
activerecord (5.1.4) lib/active_record/attribute_methods/dirty.rb:35:in `save'
activerecord (5.1.4) lib/active_record/transactions.rb:308:in `block (2 levels) in save'
activerecord (5.1.4) lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status'
activerecord (5.1.4) lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
activerecord (5.1.4) lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
/Users/omar/.rbenv/versions/2.3.0/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
activerecord (5.1.4) lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
activerecord (5.1.4) lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
activerecord (5.1.4) lib/active_record/transactions.rb:210:in `transaction'
activerecord (5.1.4) lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
activerecord (5.1.4) lib/active_record/transactions.rb:308:in `block in save'
activerecord (5.1.4) lib/active_record/transactions.rb:323:in `rollback_active_record_state!'
activerecord (5.1.4) lib/active_record/transactions.rb:307:in `save'
activerecord (5.1.4) lib/active_record/suppressor.rb:42:in `save'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/active_record_prepend.rb:16:in `block in save'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/transaction.rb:917:in `with_database_metric_name'
newrelic_rpm (5.0.0.342) lib/new_relic/agent.rb:618:in `with_database_metric_name'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/active_record_prepend.rb:15:in `save'
activeadmin (1.2.1) lib/active_admin/resource_controller/data_access.rb:155:in `block in save_resource'
activesupport (5.1.4) lib/active_support/core_ext/object/try.rb:17:in `public_send'
activesupport (5.1.4) lib/active_support/core_ext/object/try.rb:17:in `try!'
activesupport (5.1.4) lib/active_support/core_ext/object/try.rb:6:in `try'
activeadmin (1.2.1) lib/active_admin/callbacks.rb:79:in `block (2 levels) in define_active_admin_callbacks'
activeadmin (1.2.1) lib/active_admin/resource_controller/data_access.rb:154:in `save_resource'
activeadmin (1.2.1) lib/active_admin/resource_controller/data_access.rb:144:in `block in create_resource'
activesupport (5.1.4) lib/active_support/core_ext/object/try.rb:17:in `public_send'
activesupport (5.1.4) lib/active_support/core_ext/object/try.rb:17:in `try!'
activesupport (5.1.4) lib/active_support/core_ext/object/try.rb:6:in `try'
activeadmin (1.2.1) lib/active_admin/callbacks.rb:79:in `block (2 levels) in define_active_admin_callbacks'
activeadmin (1.2.1) lib/active_admin/resource_controller/data_access.rb:143:in `create_resource'
inherited_resources (1.8.0) lib/inherited_resources/actions.rb:33:in `create'
actionpack (5.1.4) lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
actionpack (5.1.4) lib/abstract_controller/base.rb:186:in `process_action'
actionpack (5.1.4) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (5.1.4) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (5.1.4) lib/active_support/callbacks.rb:131:in `run_callbacks'
actionpack (5.1.4) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (5.1.4) lib/action_controller/metal/rescue.rb:20:in `process_action'
logstasher (1.2.2) lib/logstasher/rails_ext/action_controller/metal/instrumentation.rb:34:in `block in process_action'
activesupport (5.1.4) lib/active_support/notifications.rb:166:in `block in instrument'
activesupport (5.1.4) lib/active_support/notifications/instrumenter.rb:21:in `instrument'
activesupport (5.1.4) lib/active_support/notifications.rb:166:in `instrument'
logstasher (1.2.2) lib/logstasher/rails_ext/action_controller/metal/instrumentation.rb:21:in `process_action'
actionpack (5.1.4) lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
activerecord (5.1.4) lib/active_record/railties/controller_runtime.rb:22:in `process_action'
actionpack (5.1.4) lib/abstract_controller/base.rb:124:in `process'
actionview (5.1.4) lib/action_view/rendering.rb:30:in `process'
actionpack (5.1.4) lib/action_controller/metal.rb:189:in `dispatch'
actionpack (5.1.4) lib/action_controller/metal.rb:253:in `dispatch'
actionpack (5.1.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
actionpack (5.1.4) lib/action_dispatch/routing/route_set.rb:31:in `serve'
actionpack (5.1.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (5.1.4) lib/action_dispatch/journey/router.rb:33:in `each'
actionpack (5.1.4) lib/action_dispatch/journey/router.rb:33:in `serve'
actionpack (5.1.4) lib/action_dispatch/routing/route_set.rb:834:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
omniauth (1.8.1) lib/omniauth/strategy.rb:190:in `call!'
omniauth (1.8.1) lib/omniauth/strategy.rb:168:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/rack/agent_hooks.rb:30:in `traced_call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/rack/browser_monitoring.rb:32:in `traced_call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
http_accept_language (2.1.1) lib/http_accept_language/middleware.rb:14:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
warden (1.2.7) lib/warden/manager.rb:36:in `block in call'
warden (1.2.7) lib/warden/manager.rb:35:in `catch'
warden (1.2.7) lib/warden/manager.rb:35:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
rack (2.0.4) lib/rack/etag.rb:25:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
rack (2.0.4) lib/rack/conditional_get.rb:38:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
rack (2.0.4) lib/rack/head.rb:12:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
rack (2.0.4) lib/rack/session/abstract/id.rb:232:in `context'
rack (2.0.4) lib/rack/session/abstract/id.rb:226:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/cookies.rb:613:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
activerecord (5.1.4) lib/active_record/migration.rb:556:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
activesupport (5.1.4) lib/active_support/callbacks.rb:97:in `run_callbacks'
actionpack (5.1.4) lib/action_dispatch/middleware/callbacks.rb:24:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/executor.rb:12:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
web-console (3.5.1) lib/web_console/middleware.rb:135:in `call_app'
web-console (3.5.1) lib/web_console/middleware.rb:28:in `block in call'
web-console (3.5.1) lib/web_console/middleware.rb:18:in `catch'
web-console (3.5.1) lib/web_console/middleware.rb:18:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
railties (5.1.4) lib/rails/rack/logger.rb:36:in `call_app'
railties (5.1.4) lib/rails/rack/logger.rb:24:in `block in call'
activesupport (5.1.4) lib/active_support/tagged_logging.rb:69:in `block in tagged'
activesupport (5.1.4) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (5.1.4) lib/active_support/tagged_logging.rb:69:in `tagged'
railties (5.1.4) lib/rails/rack/logger.rb:24:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
request_store (1.4.0) lib/request_store/middleware.rb:19:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/request_id.rb:25:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
rack (2.0.4) lib/rack/method_override.rb:22:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
rack (2.0.4) lib/rack/runtime.rb:22:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
activesupport (5.1.4) lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
actionpack (5.1.4) lib/action_dispatch/middleware/executor.rb:12:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
rack (2.0.4) lib/rack/sendfile.rb:111:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
railties (5.1.4) lib/rails/engine.rb:522:in `call'
newrelic_rpm (5.0.0.342) lib/new_relic/agent/instrumentation/middleware_tracing.rb:92:in `call'
puma (3.10.0) lib/puma/configuration.rb:225:in `call'
puma (3.10.0) lib/puma/server.rb:605:in `handle_request'
puma (3.10.0) lib/puma/server.rb:437:in `process_client'
puma (3.10.0) lib/puma/server.rb:301:in `block in run'
puma (3.10.0) lib/puma/thread_pool.rb:120:in `block in spawn_thread'

If I comment out the rewind in s3.rb at line 392, everything works fine and the upload completes properly. Interestingly too, if I modify my processor for testing with the following, it works too (the file passed as an argument to open is the same file generated by the micro-service). The problem seems to happen only when I download a file form my service.

module Paperclip
  class Convertion < Processor

    def initialize(file, options = {}, attachment = nil)
      super
      @file = file
      @target_format = options[:format]
      @basename = File.basename(@file.path, File.extname(@file.path))
    end

    # Main method for the processor, reference :
    # https://github.com/thoughtbot/paperclip#custom-attachment-processors
    #
    # @return [File] The processed file
      return_file = Tempfile.new([@basename, @target_format])
      return_file.binmode
      return_file.write(open("/Users/foobar/Downloads/1024.png").read)
      return_file
    end
  end
end

So... the questions are :

  1. Is this file.rewind in s3.rb at line 392 really necessary ? I am unsure on why it has been placed in there.

  2. Why do you think the files created through a download are closed but not the ones created through a local open. FYI, i have tried the second piece of code with multiple files with both open getting a local and a remote file with always the local files working and remote ones bugging.
@MadeInSyria
Copy link
Author

So after further investigation, I have found my issue. I am still unsure on why it caused the problem but I will share just in case someone has the same issue.

has_attached_file :image, styles: { png: { format: "png" }, processors: [:convertion]

:format is a reserved keyword, changing it to target_format fixed the issue. Again, no idea how that impacts the file streaming... but it does.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@MadeInSyria