Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

supermarket onsite giving errors when downloading #1449

Closed
TD-4242 opened this issue Oct 10, 2016 · 19 comments
Closed

supermarket onsite giving errors when downloading #1449

TD-4242 opened this issue Oct 10, 2016 · 19 comments

Comments

@TD-4242
Copy link

TD-4242 commented Oct 10, 2016 •
edited by robbkidd
Loading

getting the following error on the supermarket server

2016-10-10_21:41:08.79020 F, [2016-10-10T21:41:08.790015 #9000] FATAL -- :
2016-10-10_21:41:08.79026 ActionView::Template::Error (undefined method 'foodcritic_failure' for #<CookbookVersion:0x0000000a6fcfe8>):
2016-10-10_21:41:08.79027      6: json.file api_v1_cookbook_version_download_url(cookbook_version.cookbook, cookbook_version)
2016-10-10_21:41:08.79028      7: json.quality do
2016-10-10_21:41:08.79029      8:   json.foodcritic do
2016-10-10_21:41:08.79029      9:     json.failed @cookbook.foodcritic_failure
2016-10-10_21:41:08.79030     10:     json.feedback @cookbook.foodcritic_feedback
2016-10-10_21:41:08.79031     11:   end
2016-10-10_21:41:08.79031     12:   json.collaborator do
2016-10-10_21:41:08.79032   app/models/cookbook.rb:104:in 'foodcritic_failure'
2016-10-10_21:41:08.79033   app/views/api/v1/cookbook_versions/_cookbook_version.json.jbuilder:9:in 'block (2 levels) in _app_views_api_v__cookbook_versions__cookbook_version_json_jbuilder___459650076470880843_74145800'
2016-10-10_21:41:08.79033   app/views/api/v1/cookbook_versions/_cookbook_version.json.jbuilder:8:in 'block in _app_views_api_v__cookbook_versions__cookbook_version_json_jbuilder___459650076470880843_74145800'
2016-10-10_21:41:08.79034   app/views/api/v1/cookbook_versions/_cookbook_version.json.jbuilder:7:in `_app_views_api_v__cookbook_versions__cookbook_version_json_jbuilder___459650076470880843_74145800'
2016-10-10_21:41:08.79037   app/views/api/v1/cookbook_versions/show.json.jbuilder:1:in '_app_views_api_v__cookbook_versions_show_json_jbuilder__1257229507425499343_74105600'
2016-10-10_21:41:08.79037
2016-10-10_21:41:08.79038

Getting this from the client side trying to download our internal java wrapper cookbook

E, [2016-10-10T14:41:08.795612 #32070] ERROR -- : Actor crashed!
Berkshelf::CommunitySiteError: An unexpected error occurred retrieving 'dm_java' (0.1.5) from the cookbook site at ''.
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/berkshelf-5.1.0/lib/berkshelf/community_rest.rb:124:in `find'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/berkshelf-5.1.0/lib/berkshelf/community_rest.rb:100:in `download'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/berkshelf-5.1.0/lib/berkshelf/downloader.rb:61:in `try_download'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/berkshelf-5.1.0/lib/berkshelf/downloader.rb:35:in `block in download'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/berkshelf-5.1.0/lib/berkshelf/downloader.rb:34:in `each'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/berkshelf-5.1.0/lib/berkshelf/downloader.rb:34:in `download'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/berkshelf-5.1.0/lib/berkshelf/installer.rb:105:in `install'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:63:in `dispatch'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
        /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
An unexpected error occurred retrieving 'dm_java' (0.1.5) from the cookbook site at ''.

tried rolling back to 2.8.25 and getting the same messages.
@nellshamrell
Copy link
Contributor

Hi there! What version of Supermarket are you currently running? And what was the maximum version you were running before rolling back?

@TD-4242
Copy link
Author

TD-4242 commented Oct 10, 2016

I've been running the supermarket-omnibus-cookbook for a while and it has upgraded packages for us over the last year:

supermarket_2.5.2-1_amd64.deb  supermarket_2.7.3-1_amd64.deb  supermarket_2.8.15-1_amd64.deb  supermarket_2.8.29-1_amd64.deb
supermarket_2.6.0-1_amd64.deb  supermarket_2.7.4-1_amd64.deb  supermarket_2.8.2-1_amd64.deb   supermarket_2.8.3-1_amd64.deb
supermarket_2.6.1-1_amd64.deb  supermarket_2.8.0-1_amd64.deb  supermarket_2.8.25-1_amd64.deb
supermarket_2.7.2-1_amd64.deb  supermarket_2.8.1-1_amd64.deb  supermarket_2.8.27-1_amd64.deb

it started this morning, I'm guessing when 2.8.29 hit?

We did finally get things working again about 5 minutes ago. The database schema was missing fields. the columns in cookbook_versions table were missing foodcritic_failure, foodcritic_feedback, collaborator_feedbackand collaborator_failure.

We are functional now on 2.8.25-1 and will hold off here for the night.

@nellshamrell
Copy link
Contributor

There indeed was a large bug with 2.8.29, which involved a migration. Apologies for the issues! We will be doing a post-mortum very soon!

@chrisgit
Copy link

@nellshamrell Believe this is the same issue we are having.

We built a Private Supermarket using supermarket-omnibus-cookbook and have the ['supermarket_omnibus']['package_version'] set to latest.

This morning we had a Berkshelf failure on the client side and the Supermarket cookbook has been failing since around 19:00 hours last night.

We re-started the Supermarket services which fixed the Berkshelf issue but we still have an issue with the Supermarket cookbook.

The supermarket-omnibus cookbook we are using is version is 1.4.1 and unfortunately cannot downgrade the version of supermarket

* yum_package[supermarket] action upgradewarning: /var/chef/cache/supermarket-2.8.27-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
supermarket 2.8.27-1.el6
supermarket 2.8.29-1.el6
warning: /var/chef/cache/supermarket-2.8.27-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
        package supermarket-2.8.29-1.el6.x86_64 (which is newer than supermarket-2.8.27-1.el6.x86_64) is already installed

        ================================================================================
        Error executing action `upgrade` on resource 'yum_package[supermarket]'
        ================================================================================

        Mixlib::ShellOut::ShellCommandFailed
        ------------------------------------
        Expected process to exit with [0], but received '2'
        ---- Begin output of rpm  -U /var/chef/cache/supermarket-2.8.27-1.el6.x86_64.rpm ----
        STDOUT:
        STDERR: warning: /var/chef/cache/supermarket-2.8.27-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
                package supermarket-2.8.29-1.el6.x86_64 (which is newer than supermarket-2.8.27-1.el6.x86_64) is already installed
        ---- End output of rpm  -U /var/chef/cache/supermarket-2.8.27-1.el6.x86_64.rpm ----
        Ran rpm  -U /var/chef/cache/supermarket-2.8.27-1.el6.x86_64.rpm returned 2

        Resource Declaration:
        ---------------------
        # In /var/chef/cache/cookbooks/chef-ingredient/libraries/default_handler.rb

         51:       package new_resource.product_name do
         52:         action action_name
         53:         package_name ingredient_package_name
         54:         options new_resource.options
         55:         source local_path || new_resource.package_source
         56:         provider value_for_platform_family(
         57:           'debian'  => Chef::Provider::Package::Dpkg,
         58:           'rhel'    => Chef::Provider::Package::Rpm,
         59:           'windows' => Chef::Provider::Package::Windows
         60:         )
         61:         if new_resource.product_name == 'chef'
         62:           # We define this resource in ChefIngredientProvider
         63:           notifies :run, 'ruby_block[stop chef run]', :immediately
         64:         end
         65:       end

        Compiled Resource:
        ------------------
        # Declared in /var/chef/cache/cookbooks/chef-ingredient/libraries/default_handler.rb:51:in `configure_from_source_package'

        yum_package("supermarket") do
          package_name "supermarket"
          provider Chef::Provider::Package::Rpm
          action [:upgrade]
          retries 0
          retry_delay 2
          default_guard_interpreter :default
          declared_type :package
          cookbook_name "supermarket-omnibus-cookbook"
          version "2.8.27-1.el6"
          source "/var/chef/cache/supermarket-2.8.27-1.el6.x86_64.rpm"
        end

Should we manually downgrade the version of Supermarket?

@nellshamrell
Copy link
Contributor

@chrisgit to get you back and working, I do suggest manually downgrading your installation of Supermarket. Adding an issue to the Supermarket omnibus cookbook to allow downgrading.

You many need to add these fields back to the cookbook_versions table after the downgrade is complete foodcritic_failure, foodcritic_feedback, collaborator_feedback, and collaborator_failure.

Again, we apologize for this issue, and are working to remediate it right now.

@nellshamrell nellshamrell mentioned this issue Oct 11, 2016
Open
@chrisgit
Copy link

@nellshamrell Thank you for your help.

Apologies but I'm not sure how to add foodcritic_failure, foodcritic_feedback, collaborator_feedback, and collaborator_failure to the cookbook_versions table.

@nellshamrell
Copy link
Contributor

Let me find out the best way to do so and I will follow up with you shortly :)

@TD-4242 - what method did you use?

@nellshamrell
Copy link
Contributor

@chrisgit - one additional question - are you using an local DB on your Supermarket server or using a separate DB like Amazon RDS?

@chrisgit
Copy link

Hello @nellshamrell . All help and advice is gratefully received. Our Private Supermarket is using a local DB.

@nellshamrell
Copy link
Contributor

@chrisgit I am currently building an environment to test a solution and will update you hourly until it is complete.

@chrisgit
Copy link

Thank you, I'm out tonight and at Chef Summit London tomorrow but a solution is probably helpful to others who also upgraded their Supermarket installations.

@nellshamrell
Copy link
Contributor

Ah! I see that it is getting late in London, so I won't make a lot of noise by updating you hourly, but will continue to update periodically throughout the day.

@TD-4242
Copy link
Author

TD-4242 commented Oct 11, 2016

log into the postgres database as supermarket

\c supermarket;
ALTER TABLE cookbook_versions ADD COLUMN "foodcritic_failure" BOOLEAN;
ALTER TABLE cookbook_versions ADD COLUMN "collaborator_failure" BOOLEAN;
ALTER TABLE cookbook_versions ADD COLUMN "foodcritic_feedback" TEXT;
ALTER TABLE cookbook_versions ADD COLUMN "collaborator_feedback" TEXT;

@specialkevin
Copy link

Is there a proposed solution for fixing this problem currently? Should I login to the supermarket postgres database and run the commands posted by @TD-4242.

@nellshamrell
Copy link
Contributor

@specialkevin @chrisgit Yes, please use the login to the supermarket postgres database and run the commands as mentioned by @TD-4242. That will get you working and unblocked for now. I am currently working on a new version which will fix this error and it will be released within the next few days.

@ton31337
Copy link

I confirm, that altering tables solve this issue temporary. Waiting for next release.

@chrisgit
Copy link

@nellshamrell Thank you, we will attempt roll back today

@robbkidd
Copy link
Contributor

A fix has been built and is available as v2.8.30 in our current channel. It's been pretty thoroughly vetted and is running happily on our Staging instance. We're holding off releasing it to stable until next week because Friday Releases are Unfun™. I would recommend trying 2.8.30 before the effort of going through a roll back. 2.8.30 does not contain destructive changes, only some guards in the generation of the JSON output for the API endpoint that broke in 2.8.29.

@nellshamrell
Copy link
Contributor

@here we have released 2.8.30, which includes a fix for this issue https://github.com/chef/supermarket/releases/tag/2.8.30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@specialkevin @robbkidd @nellshamrell @ton31337 @TD-4242 @chrisgit