New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running private supermarket server on local network #425

Closed
sbotman opened this Issue Jun 12, 2014 · 14 comments

Comments

Projects
None yet
6 participants
@sbotman

sbotman commented Jun 12, 2014

The current setup of the supermarket is currently dependent on the OCID api which is (still) closed source. We would like to run the supermarket setup local need some other kind of authentication option.

Discussion started on gitter on how to handle this and currently it seems that some export and import is needed from the current private-chef server into the supermarket server (suggestion of @bcobb)

@cwebberOps suggested to create this issue ticket.

@bcobb

This comment has been minimized.

Contributor

bcobb commented Jun 12, 2014

Just to nip any potential misunderstanding in the bud, I didn't mean to suggest that export/import from the private chef server is necessary (it might be, but I genuinely don't know).

Rather, what I was trying to suggest was something a little more generic. From a rails console, run something like this:

user = User.create!(
  email: 'sbotman@example.com',
  roles: 'admin',
  public_key: "-----BEGIN CERTIFICATE-----" # this should be a full public key
)
Account.create!(
  provider: 'chef_oauth2',
  uid: 'sbotman',
  username: 'sbotman',
  user: user,
  oauth_token: 'sbotman'
)

With this User and Account record in place, it should be possible to use knife in conjunction with knife-supermarket to upload cookbooks to your internal Supermarket. Specifically (if I understand knife correctly) your knife.rb configuration will need to specify node_name as "sbotman" and client_key as the private key corresponding to the User record's public_key.


A more drastic way to solve the problem would be to replace this block of code with User.first!. However, this should be considered a short-term solution at best.

EDIT: Replaced the ssh-rsa public key snippet with a more indicative snippet.

@sbotman

This comment has been minimized.

sbotman commented Jun 12, 2014

Almost got it, have done the steps like you pointed out and see the users in the database.
But while "sharing" a cookbook I got the following:

knife supermarket share dummy sbp -VV
DEBUG: No chefignore file found at /home/sbotman/chefignore no files will be ignored
DEBUG: No chefignore file found at /home/sbotman/cookbooks/chefignore no files will be ignored
INFO: Validating ruby files
DEBUG: Ruby file /home/sbotman/cookbooks/dummy/metadata.rb is unchanged, skipping syntax check
DEBUG: Ruby file /home/sbotman/cookbooks/dummy/recipes/default.rb is unchanged, skipping syntax check
INFO: Validating templates
INFO: Syntax OK
DEBUG: Staging at /tmp/chef-dummy-build20140612-23563-1vwh7l2
DEBUG: Staging /home/sbotman/cookbooks/dummy/recipes/default.rb to /tmp/chef-dummy-build20140612-23563-1vwh7l2/dummy/recipes/default.rb
DEBUG: Staging /home/sbotman/cookbooks/dummy/metadata.rb to /tmp/chef-dummy-build20140612-23563-1vwh7l2/dummy/metadata.rb
DEBUG: Staging /home/sbotman/cookbooks/dummy/README.md to /tmp/chef-dummy-build20140612-23563-1vwh7l2/dummy/README.md
DEBUG: Staging /home/sbotman/cookbooks/dummy/CHANGELOG.md to /tmp/chef-dummy-build20140612-23563-1vwh7l2/dummy/CHANGELOG.md
DEBUG: Generating metadata
Generating metadata for dummy from /tmp/chef-dummy-build20140612-23563-1vwh7l2/dummy/metadata.rb
DEBUG: Generated /tmp/chef-dummy-build20140612-23563-1vwh7l2/dummy/metadata.json
DEBUG: Temp cookbook directory is "/tmp/chef-dummy-build20140612-23563-1vwh7l2"
Making tarball dummy.tgz
DEBUG: Executing tar -czf dummy.tgz dummy
DEBUG: ---- Begin output of tar -czf dummy.tgz dummy ----
DEBUG: STDOUT:
DEBUG: STDERR:
DEBUG: ---- End output of tar -czf dummy.tgz dummy ----
DEBUG: Ran tar -czf dummy.tgz dummy returned 0
DEBUG: Signing: method: post, path: /api/v1/cookbooks, file: #<File:0x00000000c0dc58>, User-id: sbotman, Timestamp: 2014-06-12T16:56:34Z
ERROR: Error uploading cookbook dummy to the Opscode Cookbook Site: lexical error: invalid char in json text.
                                   500 Internal Server Error If you ar
                     (right here) ------^
. Increase log verbosity (-VV) for more information.
DEBUG:
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/yajl-ruby-1.2.0/lib/yajl.rb:37:in `parse'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/yajl-ruby-1.2.0/lib/yajl.rb:37:in `parse'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.4/lib/chef/json_compat.rb:56:in `from_json'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-supermarket-0.0.3/lib/chef/knife/supermarket_share.rb:51:in `do_upload'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.4/lib/chef/knife/cookbook_site_share.rb:67:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.4/lib/chef/knife.rb:492:in `run_with_pretty_exceptions'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.4/lib/chef/knife.rb:174:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.4/lib/chef/application/knife.rb:135:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.4/bin/knife:25:in `<top (required)>'
/usr/bin/knife:23:in `load'
/usr/bin/knife:23:in `<main>'

So no authentication errors but something with invalid char in json..
I'm searching for the error but maybe you have seen this before..

@bcobb

This comment has been minimized.

Contributor

bcobb commented Jun 12, 2014

It looks like Supermarket returned a 500 with a non-JSON body. Do the rails server logs give any clues?

@sbotman

This comment has been minimized.

sbotman commented Jun 12, 2014

Hi @bcobb

Within the log I see "Error during failsafe response: ActionController::InvalidAuthenticityToken" so I assume I have uploaded the wrong key-format. What is the key length and how is the key-pair normally created? Maybe good to create a new one and see if I can make it work with that one since migration from the current private-chef server doesn't seem to work.

Thanks Sander

@cwebberOps

This comment has been minimized.

Contributor

cwebberOps commented Jun 13, 2014

Hey @smith and @cnunciato I would love your input on what this might look like behind the firewall...

@tristanoneil

This comment has been minimized.

Contributor

tristanoneil commented Jun 13, 2014

@sbotman Hmm ActionController::InvalidAuthenticityToken actually looks like it has to do with Rails CSFR protection which we're skipping in the API so this is quite an odd error. I think we need to discuss this a bit more and come up with a real solution for running Supermarket privately not just work arounds. To answer your question about key formats though it should be the same key format that chef server uses. @cwebberOps we should probably discuss this during planning next week.

@sbotman

This comment has been minimized.

sbotman commented Jun 13, 2014

If I look in the database of our current private-chef server, the public-keys are stored as
-----BEGIN CERTIFICATE-----\nMIIDMz...blabla......-----END CERTIFICATE-----
So plain migration doesn't seem to work because as I understand from the comments here above the key should be in the "ssh-rsa ..." format.

@tristanoneil

This comment has been minimized.

Contributor

tristanoneil commented Jun 13, 2014

@sbotman I think the specifics of the above comment may be incorrect. Looking at the public keys that exist in Supermarket currently they're in the same format that Chef server stores them -----BEGIN CERTIFICATE-----\nMIIDMz...blabla......-----END CERTIFICATE-----. Supermarket uses Mixlib::Authentication to digest the signed headers coming from Knife (signed using Mixlib::Authentication) which I believe Chef server uses as well.

@bcobb

This comment has been minimized.

Contributor

bcobb commented Jun 13, 2014

Sorry for the misunderstanding. I should have mentioned that the public_key value should be in the same format as your private Chef server, but didn't reference actual values to make it apparent.

Regardless, as @tristanoneil said, that error looks like a different problem than mismatched key formats.

@sbotman are you using the supermarket cookbook to run Supermarket? If not, how are you running the app?

@sbotman

This comment has been minimized.

sbotman commented Jun 13, 2014

Yes, we have used the cookbook to install the supermarket with some minor adjustments, please see: https://github.com/svanharmelen/supermarket/tree/use_community_cookbooks
The pull request is open (nr 11) and it's all running fine.

Now I have copied the key exactly and it still fails. Have tried all kind of formats, but seems that it fails all the time. Because we want to have this POC finished today, we will go another route...

@smith

This comment has been minimized.

Contributor

smith commented Jun 13, 2014

The method @bcobb described (using the Rails console to create an account and user with the public key) should work in theory. The key that's needed there is the one you can get from the Chef Management Console by going to Administration > Users > you (/organizations/your-org/users/you) and copying the Public Key.

Our long-term plan is to open source oc-id and ship it with the Chef server (open source and Enterprise.)

I wouldn't recommend waiting on that, since we don't have a definite timeline on when that is going to happen (it's only running in Hosted Enterprise Chef right now.)

A better medium-term solution might be to run your own OAuth 2 service using Doorkeeper that uses the omniauth-chef strategy. This is essentially all that oc-id is.

Obviously that's not an ideal solution, but right now it's the closest thing possible to get Supermarket running on your private infrastructure and working like the hosted version does.

Hope that helps a little, and let me know if you have any more questions.

@irvingpop

This comment has been minimized.

Contributor

irvingpop commented Jul 17, 2014

@smith @cwebberOps @bcobb Great work on this! We're starting to get pinged by customers who want to try this out and plan internal (behind-the-firewall) usage of Supermarket. Could you please help us make the experience a little bit more delightful for them?

To recap my understanding of the discussion so far:

  • Supermarket currently uses the omniauth-chef-oauth2 strategy for OmniAuth, with https://id.opscode.com hardcoded in it
    • id.opscode.com runs oc-id, which is closed-source with no clear timeline for open sourcing it
  • The current Minimum Viable Authentication is really to use the rails console to manually add users as described above
  • Some point Soon(tm) id.opscode.com will have support for registering applications, so you can set your internal Supermarket server to authenticate against the Community site
  • Later On(tm) you'll be able to run your own oc-id (once open sourced) to authenticate against your Enterprise Chef server

It would be super awesome if we clarified a few points for our new Supermarket users:

  • Adding examples here and in the wiki for using the rails console to manually add users
  • Adding examples for configuring Supermarket to use your own OAuth2 service
@smith

This comment has been minimized.

Contributor

smith commented Jul 20, 2014

@irvingpop you're pretty close here. to clarify:

  • id.opscode.com runs oc-id, which is closed-source with no clear timeline for open sourcing it

Up until now there has been no clear timeline, but given the need for it, open-sourcing it is likely going to happen in the very near future.

  • Some point Soon(tm) id.opscode.com will have support for registering applications, so you can set your internal Supermarket server to authenticate against the Community site

I don't think we have any plans to do this, though I don't see a problem with it in principle.

  • Later On(tm) you'll be able to run your own oc-id (once open sourced) to authenticate against your Enterprise Chef server

That's correct, but again, it's more likely Sooner On. I'm deliberately using weasel words to not commit that it's going to be open-sourced because we haven't made that decision, but it is the decision of the Front End team, and the only thing standing in the way is if somebody has valid objections to it happening.

Later On, we'll ship oc-id with Chef server, so people running an internal Supermarket won't need to stand up an oc-id and it can be used for auth with other services (Manage, Analytics, etc.)

@smith

This comment has been minimized.

Contributor

smith commented Aug 2, 2014

This should now be possible since we've open sourced oc-id.

Once you have Supermarket running locally, the best place to start would be the Chef Authentication page on the Supermarket wiki.

If you have any trouble getting it running, we should be able to help.

@smith smith closed this Aug 2, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment