bolt_task agent is unable to download tasks containing multiple files

[jay7x]

I did bolt task using Ruby TaskHelper module. Here are task details:

$ mco tasks --detail spd::query_exporter
Retrieving task metadata for task spd::query_exporter from the Puppet Server

spd::query_exporter - Query values from prometheus exporter

Task Parameters:
  metrics                        Metrics to fetch (Optional[Variant[String[1], Array[String]]])
  url                            Exporter URL to query (String[1])

Task Files:
  query_exporter.rb              2152 bytes
  ruby_task_helper/files/task_helper.rb 1619 bytes
  spd/files/task_helper.rb       503 bytes

When trying to execute this task I hit into following issue:

 Could not download task file: Errno::ENOENT: No such file or directory @ rb_file_s_rename - (/tmp/tasks_ruby_task_helperfilestask_helper.rb20191119-108923-12rsdll, /opt/puppetlabs/mcollective/tasks-cache/e07ab8a82c450803abc67439f0f3145e7e80bcf9ee038f7399c7ebc55a360cae/ruby_task_helper/files/task_helper.rb)

Here is choria-mcorpc.log content related:

I, [2019-11-19T19:04:33.665514 #108923]  INFO -- : config.rb:164:in `loadconfig' The Marionette Collective version 2.20.7 started by /usr/bin/choria_mcollective_agent_compat.rb using config file /etc/
puppetlabs/mcollective/choria-shim.cfg
I, [2019-11-19T19:04:34.460804 #108923]  INFO -- : tasks_support.rb:689:in `download_files' Downloading 3 task files
E, [2019-11-19T19:04:34.662913 #108923] ERROR -- : tasks_support.rb:705:in `rescue in block in download_files' Could not download task file: Errno::ENOENT: No such file or directory @ rb_file_s_rename - (/tmp/tasks_ruby_task_helperfilestask_helper.rb20191119-108923-nrh1xu, /opt/puppetlabs/mcollective/tasks-cache/e07ab8a82c450803abc67439f0f3145e7e80bcf9ee038f7399c7ebc55a360cae/ruby_task_helper/files/task_helper.rb)
E, [2019-11-19T19:04:34.841060 #108923] ERROR -- : tasks_support.rb:705:in `rescue in block in download_files' Could not download task file: Errno::ENOENT: No such file or directory @ rb_file_s_rename - (/tmp/tasks_ruby_task_helperfilestask_helper.rb20191119-108923-12rsdll, /opt/puppetlabs/mcollective/tasks-cache/e07ab8a82c450803abc67439f0f3145e7e80bcf9ee038f7399c7ebc55a360cae/ruby_task_helper/files/task_helper.rb)
E, [2019-11-19T19:04:34.941428 #108923] ERROR -- : agent.rb:108:in `rescue in handlemsg' bolt_tasks#download failed: RuntimeError: Could not download task file: Errno::ENOENT: No such file or directory @ rb_file_s_rename - (/tmp/tasks_ruby_task_helperfilestask_helper.rb20191119-108923-12rsdll, /opt/puppetlabs/mcollective/tasks-cache/e07ab8a82c450803abc67439f0f3145e7e80bcf9ee038f7399c7ebc55a360cae/ruby_task_helper/files/task_helper.rb)
E, [2019-11-19T19:04:34.941578 #108923] ERROR -- : agent.rb:109:in `rescue in handlemsg' /opt/puppetlabs/mcollective/plugins/mcollective/util/tasks_support.rb:711:in `rescue in block in download_files'
        /opt/puppetlabs/mcollective/plugins/mcollective/util/tasks_support.rb:696:in `block in download_files'
        /opt/puppetlabs/mcollective/plugins/mcollective/util/tasks_support.rb:691:in `each'
        /opt/puppetlabs/mcollective/plugins/mcollective/util/tasks_support.rb:691:in `download_files'
        /opt/puppetlabs/mcollective/plugins/mcollective/agent/bolt_tasks.rb:18:in `block in <class:Bolt_tasks>'
        /opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/choria-mcorpc-support-2.20.7/lib/mcollective/rpc/agent.rb:86:in `handlemsg'
        /usr/bin/choria_mcollective_agent_compat.rb:150:in `block in dispatch'
        /opt/puppetlabs/puppet/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'
        /opt/puppetlabs/puppet/lib/ruby/2.5.0/timeout.rb:33:in `block in catch'
        /opt/puppetlabs/puppet/lib/ruby/2.5.0/timeout.rb:33:in `catch'
        /opt/puppetlabs/puppet/lib/ruby/2.5.0/timeout.rb:33:in `catch'
        /opt/puppetlabs/puppet/lib/ruby/2.5.0/timeout.rb:108:in `timeout'
        /usr/bin/choria_mcollective_agent_compat.rb:145:in `dispatch'
        /usr/bin/choria_mcollective_agent_compat.rb:174:in `<main>'

[ripienaar]

interesting, yeah I think when I did the task support the specification didnt yet allow multiple files in a task

[smortex]

I'm hitting this with this module: odoo.

The choria-audit log shows (reformatted for readability):

[
  {
    "filename": "database_delete.sh",
    "sha256": "27cf85fc65b33bd3492e0188533d4a25d67cd810d91f61b4baf6e89cc675539a",
    "size_bytes": 140,
    "uri": {
      "path": "/puppet/v3/static_file_content/modules/odoo/tasks/database_delete.sh",
      "params": {
        "environment": "production",
        "code_id": "cf812ee55e84bf63a0b8ee4a08b2c3fad26dc313"
      }
    }
  },
  {
    "filename": "odoo/files/database-delete.py",
    "sha256": "deffdfa89fac5e02d689d26ae43c933e47cb10ea12d56dd6c72184043bc56f25",
    "size_bytes": 96,
    "uri": {
      "path": "/puppet/v3/static_file_content/modules/odoo/files/database-delete.py",
      "params": {
        "environment": "production",
        "code_id": "cf812ee55e84bf63a0b8ee4a08b2c3fad26dc313"
      }
    }
  }
]

database_delete.sh is the script in the tasks directory, it always worked. The other file — odoo/files/database-delete.py — was added more recently while improving the module. It is in the files directory and is listed as a "dependency" in tasks/database_delete.json. The size and sha256 or the correct ones. Note however how one file has no path information and the other has module/directory/filename.

In the puppetserver log, I can see this file being successfully transferred:

2607:5300:201:3100:0:0:0:ec8 - - [17/Dec/2020:20:32:34 -0500] "GET /puppet/v3/static_file_content/modules/odoo/files/database-delete.py?environment=production&code_id=cf812ee55e84bf63a0b8ee4a08b2c3fad26dc313 HTTP/1.1" 200 96 "-" "Choria version 0.19.0 http://choria.io" 6 - 

More investigation needed (next week probably), I suspect the file is downloaded to a "wrong" location which cause problem after.

[ripienaar]

Ok thanks. I think multiple files was added to the spec after I implemented the bolt support. I anticipated they would support multiple files but behaviour was undefined then no doubt we have some bugs around there to fix.

[ripienaar]

thank you @smortex