New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Knife download all, then upload all not working #7949

Open
jeremysells opened this Issue Nov 18, 2018 · 2 comments

Comments

Projects
None yet
1 participant
@jeremysells

jeremysells commented Nov 18, 2018

Hi, Hopefully this is the right place.

Description

knife download / and knife upload / is not working (self hosted chef server). I have tried deleting everything (chef-repo and server files) and I get the same issue. Ideally I was hoping to drive my Chef Server with a GitOps like approach - i.e. the git repo is authoritative (but using my CI server to push changes to my Chef Server). I suspect I need to change my approach but thought I would record this anyway. However should a download of all and upload work (like what I am doing) ?

Chef Version

Chef-server=12.18.14 (I am using the Docker image quay.io/3ofcoins/chef-server:12.18.14)
Chef-dk=3.5.13 (I am using the Docker image chef/chefdk:3.5.13 with a few extra tools added)

Platform Version

All are Ubuntu:18.04 hosts (Desktop and Server) (but I am running everything in Docker)

Replication Case

On the server
Running the Chef server (in my case docker run ... quay.io/3ofcoins/chef-server:12.18.14)

$ chef-server-ctl org-create jeremyorg 'Jeremy Org'
$ chef-server-ctl user-create jeremy Jeremy Sells somebody@example.com --prompt-for-password
$ chef-server-ctl org-user-add jeremyorg jeremy --admin
User jeremy is added to admins and billing-admins group
$ chef-server-ctl grant-server-admin-permissions jeremy
The PGconn, PGresult, and PGError constants are deprecated, and will be
removed as of version 1.0.

You should use PG::Connection, PG::Result, and PG::Error instead, respectively.

Called from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/chef-server-ctl-1.0.0/plugins/server_admins.rb:158:in `setup_erchef_db'
User jeremy was added to server-admins. This user can now list, read, create, and delete users (even for orgs they are not members of) for this Chef Server.
$ chef-server-ctl reconfigure

So at this stage, I should be an admin of the server and an admin of the organisation.

On the development machine

$ cat /home/jeremy/.chef/config.rb
chef_server_url 'https://chef.example.com/organizations/jeremyorg'
client_key '/home/jeremy/.chef/jeremy.pem'
node_name 'jeremy'
cookbook_copyright 'Jeremy Sells'
cookbook_email 'somebody@example.com'
#config_log_level :debug
license 'Proprietary - All Rights Reserved'

versioned_cookbooks = true
knife[:editor] = 'nano'
knife[:repo_mode] = 'hosted_everything'
knife[:versioned_cookbooks] = true

See: Client output

Client Output

Note: The folder /app/chef-repo folder is empty

$knife download / --chef-repo-path /app/chef-repo --repo-mode hosted_everything
Created cookbooks
Created groups
Created data_bags
Created acls
Created clients
Created environments
Created containers
Created cookbook_artifacts
Created acls/clients
Created nodes
Created groups/0000000000004945a8c441be51b6c561.json
Created invitations.json
Created members.json
Created policies
Created clients/jeremyorg-validator.json
Created policy_groups
Created roles
Created environments/_default.json
Created acls/containers
Created containers/clients.json
Created acls/clients/jeremyorg-validator.json
Created groups/admins.json
Created acls/cookbook_artifacts
Created acls/cookbooks
Created acls/data_bags
Created acls/environments
Created org.json
Created acls/groups
Created acls/nodes
Created acls/policies
Created acls/policy_groups
Created acls/roles
Created containers/containers.json
Created acls/containers/clients.json
Created groups/billing-admins.json
Created acls/environments/_default.json
Created acls/organization.json
Created acls/groups/0000000000004945a8c441be51b6c561.json
Created containers/cookbook_artifacts.json
Created acls/containers/containers.json
Created groups/clients.json
Created groups/public_key_read_access.json
Created containers/cookbooks.json
Created groups/users.json
Created acls/groups/admins.json
Created acls/containers/cookbook_artifacts.json
Created containers/data.json
Created containers/environments.json
Created containers/nodes.json
Created containers/groups.json
Created acls/containers/cookbooks.json
ERROR: acls/groups/billing-admins.json failed to read: HTTP error reading: 403 "Forbidden"
Created containers/policies.json
Created containers/policy_groups.json
Created containers/roles.json
Created containers/sandboxes.json
Created acls/containers/data.json
Created acls/groups/clients.json
Created acls/containers/environments.json
Created acls/containers/groups.json
ERROR: acls/groups/public_key_read_access.json failed to read: HTTP error reading: 403 "Forbidden"
Created acls/containers/nodes.json
Created acls/containers/policy_groups.json
Created acls/containers/policies.json
Created acls/containers/roles.json
Created acls/containers/sandboxes.json
Created acls/groups/users.json

Note: The 2 errors above, might be important?

$ knife upload / --purge --force --chef-repo-path /app/chef-repo --repo-mode hosted_everything
/app/chef-repo$ knife upload / --purge --force --chef-repo-path /app/chef-repo --repo-mode hosted_everything -VV
INFO: Using configuration from /home/jeremy/.chef/config.rb
Updated invitations.json
Updated members.json
WARNING: environments/_default.json cannot be updated (default environment cannot be modified).
Updated org.json
Updated clients/jeremyorg-validator.json
Updated groups/0000000000004945a8c441be51b6c561.json
ERROR: containers/clients.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/containers.json failed to write: HTTP error writing: 405 "Method Not Allowed"
Updated groups/admins.json
Updated groups/billing-admins.json
Updated groups/clients.json
ERROR: containers/cookbook_artifacts.json failed to write: HTTP error writing: 405 "Method Not Allowed"
Updated groups/public_key_read_access.json
Updated acls/clients/jeremyorg-validator.json
Updated groups/users.json
ERROR: containers/cookbooks.json failed to write: HTTP error writing: 405 "Method Not Allowed"
Updated acls/containers/clients.json
ERROR: containers/data.json failed to write: HTTP error writing: 405 "Method Not Allowed"
Updated acls/groups/0000000000004945a8c441be51b6c561.json
Updated acls/environments/_default.json
ERROR: containers/environments.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/groups.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/nodes.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/policies.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/policy_groups.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/roles.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/sandboxes.json failed to write: HTTP error writing: 405 "Method Not Allowed"
Updated acls/environments/_default.json
Updated acls/containers/containers.json
Updated acls/groups/admins.json
Updated acls/containers/cookbook_artifacts.json
Updated acls/clients/jeremyorg-validator.json
Updated acls/containers/cookbooks.json
Updated acls/containers/data.json
Updated acls/containers/environments.json
Updated acls/containers/groups.json
Updated acls/groups/clients.json
Updated acls/containers/nodes.json
Updated acls/containers/policies.json
Updated acls/containers/policy_groups.json
Updated acls/containers/roles.json
Updated acls/containers/sandboxes.json
Updated acls/groups/users.json
Updated acls/containers/clients.json
Updated acls/containers/containers.json
Updated acls/containers/cookbook_artifacts.json
Updated acls/containers/cookbooks.json
Updated acls/containers/data.json
Updated acls/groups/0000000000004945a8c441be51b6c561.json
Updated acls/containers/environments.json
Updated acls/containers/groups.json
Updated acls/containers/nodes.json
ERROR: acls/groups/billing-admins.json ACLs cannot be deleted.
Updated acls/containers/policies.json
ERROR: acls/groups/public_key_read_access.json ACLs cannot be deleted.
Updated acls/containers/policy_groups.json
Updated acls/groups/admins.json
Updated acls/containers/roles.json
Updated acls/containers/sandboxes.json
Updated acls/groups/clients.json
Updated acls/groups/users.json
Traceback (most recent call last):
        10: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
         9: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:92:in `worker_loop'
         8: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:92:in `call'
         7: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in `process_one'
         6: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `process_input'
         5: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:359:in `block in copy_entries'
         4: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:389:in `copy_entries'
         3: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:49:in `write'
         2: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:49:in `each'
         1: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:50:in `block in write'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:56:in `rescue in block in write': Chef::ChefFS::FileSystem::NotFoundError (Chef::ChefFS::FileSystem::NotFoundError)
$knife upload / --purge --force --chef-repo-path /app/chef-repo --repo-mode hosted_everything -VV --concurrency 1
INFO: Using configuration from /home/jeremy/.chef/config.rb
Updated acls/clients/jeremyorg-validator.json
Updated acls/clients/jeremyorg-validator.json
Updated acls/containers/clients.json
Updated acls/containers/containers.json
Updated acls/containers/cookbook_artifacts.json
Updated acls/containers/cookbooks.json
Updated acls/containers/data.json
Updated acls/containers/environments.json
Updated acls/containers/groups.json
Updated acls/containers/nodes.json
Updated acls/containers/policies.json
Updated acls/containers/policy_groups.json
Updated acls/containers/roles.json
Updated acls/containers/sandboxes.json
Updated acls/containers/clients.json
Updated acls/containers/containers.json
Updated acls/containers/cookbook_artifacts.json
Updated acls/containers/cookbooks.json
Updated acls/containers/data.json
Updated acls/containers/environments.json
Updated acls/containers/groups.json
Updated acls/containers/nodes.json
Updated acls/containers/policies.json
Updated acls/containers/policy_groups.json
Updated acls/containers/roles.json
Updated acls/containers/sandboxes.json
Updated acls/environments/_default.json
Updated acls/environments/_default.json
Updated acls/groups/0000000000004945a8c441be51b6c561.json
Updated acls/groups/admins.json
Updated acls/groups/clients.json
Updated acls/groups/users.json
Updated acls/groups/0000000000004945a8c441be51b6c561.json
Updated acls/groups/admins.json
ERROR: acls/groups/billing-admins.json ACLs cannot be deleted.
Updated acls/groups/clients.json
ERROR: acls/groups/public_key_read_access.json ACLs cannot be deleted.
Updated acls/groups/users.json
Updated clients/jeremyorg-validator.json
ERROR: containers/clients.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/containers.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/cookbook_artifacts.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/cookbooks.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/data.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/environments.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/groups.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/nodes.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/policies.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/policy_groups.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/roles.json failed to write: HTTP error writing: 405 "Method Not Allowed"
ERROR: containers/sandboxes.json failed to write: HTTP error writing: 405 "Method Not Allowed"
WARNING: environments/_default.json cannot be updated (default environment cannot be modified).
Updated groups/0000000000004945a8c441be51b6c561.json
Updated groups/admins.json
Updated groups/billing-admins.json
Updated groups/clients.json
Updated groups/public_key_read_access.json
Updated groups/users.json
Updated invitations.json
Updated members.json
Updated org.json
Traceback (most recent call last):
        42: from /opt/chefdk/bin/knife:296:in `<main>'
        41: from /opt/chefdk/bin/knife:296:in `load'
        40: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/bin/knife:24:in `<top (required)>'
        39: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/application/knife.rb:161:in `run'
        38: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/knife.rb:220:in `run'
        37: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/knife.rb:445:in `run_with_pretty_exceptions'
        36: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/local_mode.rb:44:in `with_server_connectivity'
        35: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/knife.rb:446:in `block in run_with_pretty_exceptions'
        34: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/knife/upload.rb:75:in `run'
        33: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/knife/upload.rb:75:in `each'
        32: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/knife/upload.rb:76:in `block in run'
        31: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:140:in `copy_to'
        30: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:426:in `parallel_do'
        29: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:29:in `parallel_do'
        28: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:48:in `parallel_do'
        27: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:in `wait'
        26: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:193:in `each_with_exceptions_unordered'
        25: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in `process_one'
        24: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `process_input'
        23: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:143:in `block in copy_to'
        22: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:358:in `copy_entries'
        21: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:426:in `parallel_do'
        20: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:29:in `parallel_do'
        19: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:48:in `parallel_do'
        18: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:in `wait'
        17: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:193:in `each_with_exceptions_unordered'
        16: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in `process_one'
        15: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `process_input'
        14: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:359:in `block in copy_entries'
        13: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:358:in `copy_entries'
        12: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:426:in `parallel_do'
        11: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:29:in `parallel_do'
        10: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer.rb:48:in `parallel_do'
         9: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:in `wait'
         8: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:193:in `each_with_exceptions_unordered'
         7: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in `process_one'
         6: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `process_input'
         5: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:359:in `block in copy_entries'
         4: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system.rb:389:in `copy_entries'
         3: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:49:in `write'
         2: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:49:in `each'
         1: from /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:50:in `block in write'
/opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/chef_fs/file_system/chef_server/acl_entry.rb:56:in `rescue in block in write': Chef::ChefFS::FileSystem::NotFoundError (Chef::ChefFS::FileSystem::NotFoundError)
@jeremysells

This comment has been minimized.

jeremysells commented Nov 19, 2018

Also of note, if I add some client, I get the following error:

$ knife upload / --chef-repo-path /app/chef-repo --repo-mode hosted_everything
ERROR: acls/clients ACLs can only be updated, and can only be created when the corresponding object is created..
Created clients/Asteroid-Test-A.json
Updated groups/clients.json
Updated acls/clients/Asteroid-Test-B.json
Updated acls/nodes/Asteroid-Test-A.json
Updated acls/clients/Asteroid-Test-B.json

I guess its because its uploading alphabetically.

As expected though, if I upload again, it works without errors:

$ knife upload / --chef-repo-path /app/chef-repo --repo-mode hosted_everything
Updated groups/clients.json
Updated acls/clients/Asteroid-Test-A.json
Updated acls/clients/Asteroid-Test-A.json
@jeremysells

This comment has been minimized.

jeremysells commented Nov 19, 2018

knife upload / --purge seems to work a bit better, but it still complains about the follow files

$ knife upload / --purge
ERROR: acls/groups/billing-admins.json ACLs cannot be deleted.
ERROR: acls/groups/public_key_read_access.json ACLs cannot be deleted.
$ echo $?
1
$ echo $?
0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment