New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Impossible to upload cookbooks if non-standard port is used #50

Open
oleg-z opened this Issue Dec 23, 2014 · 20 comments

Comments

Projects
None yet
@oleg-z

oleg-z commented Dec 23, 2014

My chef-server.rb:

nginx['non_ssl_port'] = 8081
nginx['ssl_port'] = 4000

After running chef-server-ctl reconfigure I can't upload cookbook:

INFO: Uploading /var/default/chef-data/cookbooks/squeeze64-1.13.0/yum/templates/default/main.erb (checksum hex = e5ab84fe45a83c038ff442722be03dbd) to https://127.0.0.1:4000/bookshelf/organization-55800a42d41ca4067b8d9cc3f9d1ab51/checksum-e5ab84fe45a83c038ff442722be03dbd?AWSAccessKeyId=e737796ac367dbe4a94c96ad3ed439d9a3099d17&Expires=1419376039&Signature=clYmIZ0ViZA3dXgosSXDTiH9LfA%3D
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://127.0.0.1:4000/organizations/default/sandboxes/9cc3f9d1ab51f248e5ce1ccd3913b6da, retrying 1/5 in 3s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://127.0.0.1:4000/organizations/default/sandboxes/9cc3f9d1ab51f248e5ce1ccd3913b6da, retrying 2/5 in 8s                                              
INFO: HTTP Request Returned 500 Internal Server Error: internal service error                                                                                                                
ERROR: Server returned error for https://127.0.0.1:4000/organizations/default/sandboxes/9cc3f9d1ab51f248e5ce1ccd3913b6da, retrying 3/5 in 10s

Reason:
incorrect erchef template which assumes that default protocol port is used. Attempt to specify vip parameter with port (e.g. 1.1.1.1:4000) causes issue because normalize_host method parses specified string as IPv6

{chef_objects, [
                  {s3_access_key_id, "<%= node['private_chef']['bookshelf']['access_key_id'] %>"},
                  {s3_secret_key_id, "<%= node['private_chef']['bookshelf']['secret_access_key'] %>"},
                  {s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

@oleg-z oleg-z changed the title from Impossible to upload cookbooks if non-standard port are used to Impossible to upload cookbooks if non-standard port is used Dec 29, 2014

@stevendanna

This comment has been minimized.

Show comment
Hide comment
@stevendanna

stevendanna Jan 13, 2015

Member

@oleg-z Thanks for reporting this. I've confirmed the behavior you are seeing and your root cause is spot on. Below is a slightly more verbose description of what is happening for someone not as familiar with how opscode-omnibus is all tied together.

With the following placed in private-chef.rb:

nginx['non_ssl_port'] = 8081
nginx['ssl_port'] = 4000

nginx will listen on port 4000 for HTTPS connections and not the default port of 443.

During cookbook uploads, the opscode-erchef service talks to bookshelf via the s3_url in its configuration file (/var/opt/opscode/opscode-erchef/etc/app.config). This configuration file is rendered via a template(opscode-omnibus/files/private-chef-cookbooks/private-chef/templates/default/oc_erchef.config.erb), a portion of which looks like:

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

Thus, the rendered configuration file will have an s3_url like:

{s3_url, "https://private-chef.opscode.piab"},

Given this configuration, erchef will attempt to contact erchef on port 443, the default HTTPS port. Unfortunately, nothing is listening on 443, the request to bookshelf fails and erchef returns a 500 to the user.

An astute user may attempt to set bookshelf['vip'] in private-chef.rb to something like:

bookshelf['vip'] = 'private-chef.opscode.piab:4000'

Unfortunately, this does not work either, since our IPv6 handling code will mangle this into:

{s3_url, "https://[private-chef.opscode.piab:4000]"},

which also fails. If you manually edit /var/opt/opscode/opscode-erchef/etc/app.config to make the s3_url line as follows:

{s3_url, "https://private-chef.opscode.piab:4000"},

and restart opscode-erchef, cookbook uploads will be successful.

The code that interprets anything with an ":" as an IPv6 address can be found in opscode-omnibus/files/private-chef-cookbooks/private-chef/libraries/helper.rb:

  def self.normalize_host(host_part)
    # Make this simple: if ':' is detected at all, it is assumed
    # to be a valid ipv6 address. We don't do data validation at this
    # point, and ':' is only valid in an URL if it is quoted by brackets.
    if host_part =~ /:/
      "[#{host_part}]"
    else
      host_part
    end
  end

  def normalize_host(host_part)
    self.class.normalize_host(host_part)
  end

  def vip_for_uri(service)
    normalize_host(node['private_chef'][service]['vip'])
  end
Member

stevendanna commented Jan 13, 2015

@oleg-z Thanks for reporting this. I've confirmed the behavior you are seeing and your root cause is spot on. Below is a slightly more verbose description of what is happening for someone not as familiar with how opscode-omnibus is all tied together.

With the following placed in private-chef.rb:

nginx['non_ssl_port'] = 8081
nginx['ssl_port'] = 4000

nginx will listen on port 4000 for HTTPS connections and not the default port of 443.

During cookbook uploads, the opscode-erchef service talks to bookshelf via the s3_url in its configuration file (/var/opt/opscode/opscode-erchef/etc/app.config). This configuration file is rendered via a template(opscode-omnibus/files/private-chef-cookbooks/private-chef/templates/default/oc_erchef.config.erb), a portion of which looks like:

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

Thus, the rendered configuration file will have an s3_url like:

{s3_url, "https://private-chef.opscode.piab"},

Given this configuration, erchef will attempt to contact erchef on port 443, the default HTTPS port. Unfortunately, nothing is listening on 443, the request to bookshelf fails and erchef returns a 500 to the user.

An astute user may attempt to set bookshelf['vip'] in private-chef.rb to something like:

bookshelf['vip'] = 'private-chef.opscode.piab:4000'

Unfortunately, this does not work either, since our IPv6 handling code will mangle this into:

{s3_url, "https://[private-chef.opscode.piab:4000]"},

which also fails. If you manually edit /var/opt/opscode/opscode-erchef/etc/app.config to make the s3_url line as follows:

{s3_url, "https://private-chef.opscode.piab:4000"},

and restart opscode-erchef, cookbook uploads will be successful.

The code that interprets anything with an ":" as an IPv6 address can be found in opscode-omnibus/files/private-chef-cookbooks/private-chef/libraries/helper.rb:

  def self.normalize_host(host_part)
    # Make this simple: if ':' is detected at all, it is assumed
    # to be a valid ipv6 address. We don't do data validation at this
    # point, and ':' is only valid in an URL if it is quoted by brackets.
    if host_part =~ /:/
      "[#{host_part}]"
    else
      host_part
    end
  end

  def normalize_host(host_part)
    self.class.normalize_host(host_part)
  end

  def vip_for_uri(service)
    normalize_host(node['private_chef'][service]['vip'])
  end
@tokenrain

This comment has been minimized.

Show comment
Hide comment
@tokenrain

tokenrain Mar 6, 2015

The problem with this solution, as I am sure you already know, is that running chef-server-ctl reconfigure will wipe out the custom modification. Are there any plans to fix this. It seems to me that in reality Chef Server does not really support any other port than 443 in reality

tokenrain commented Mar 6, 2015

The problem with this solution, as I am sure you already know, is that running chef-server-ctl reconfigure will wipe out the custom modification. Are there any plans to fix this. It seems to me that in reality Chef Server does not really support any other port than 443 in reality

@shuoli84

This comment has been minimized.

Show comment
Hide comment
@shuoli84

shuoli84 Mar 6, 2015

I met this and it really hurts. This is a bug, right?
Either we should able to configure bookshelf port or bookshelf should return the redirect url with the same port as chef server.

shuoli84 commented Mar 6, 2015

I met this and it really hurts. This is a bug, right?
Either we should able to configure bookshelf port or bookshelf should return the redirect url with the same port as chef server.

@stevendanna stevendanna added the bug label Mar 6, 2015

@jeremiahsnapp

This comment has been minimized.

Show comment
Hide comment
@jeremiahsnapp

jeremiahsnapp Mar 24, 2015

Contributor

I also hope this gets fixed soon. In the meantime you can also manually edit the template that creates the erchef config file so the modification will survive a chef-server-ctl reconfigure.

Look around line 163 of the file /opt/opscode/embedded/cookbooks/private-chef/templates/default/oc_erchef.config.erb.

For example, if you want to set s3_url to "https://private-chef.opscode.piab:4000"

then replace the following line:

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

with this:

{s3_url, "https://private-chef.opscode.piab:4000"},

Then run chef-server-ctl reconfigure.

Contributor

jeremiahsnapp commented Mar 24, 2015

I also hope this gets fixed soon. In the meantime you can also manually edit the template that creates the erchef config file so the modification will survive a chef-server-ctl reconfigure.

Look around line 163 of the file /opt/opscode/embedded/cookbooks/private-chef/templates/default/oc_erchef.config.erb.

For example, if you want to set s3_url to "https://private-chef.opscode.piab:4000"

then replace the following line:

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

with this:

{s3_url, "https://private-chef.opscode.piab:4000"},

Then run chef-server-ctl reconfigure.

@neutralrockets

This comment has been minimized.

Show comment
Hide comment
@neutralrockets

neutralrockets Apr 15, 2015

This is not only a problem with uploading cookbooks. I have found the same sort of issue with trying to get the Chef 12 reporting working for a Chef server on a non-standard port.

For reporting, had to resort to manually editing pedant_config.rb.erb and oc_reporting.config.erb under /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/ and added

:<%= node['private_chef']['nginx']['ssl_port'] %>

so that the reconfiguration generates the port in the config files. After a reconfigure and restart, reporting seems to be working. Before this, the chef-client would take a while to run on the remote node because it would be getting "500 Internal server errors" and retrying to connect to the reporting server several times before failing.

neutralrockets commented Apr 15, 2015

This is not only a problem with uploading cookbooks. I have found the same sort of issue with trying to get the Chef 12 reporting working for a Chef server on a non-standard port.

For reporting, had to resort to manually editing pedant_config.rb.erb and oc_reporting.config.erb under /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/ and added

:<%= node['private_chef']['nginx']['ssl_port'] %>

so that the reconfiguration generates the port in the config files. After a reconfigure and restart, reporting seems to be working. Before this, the chef-client would take a while to run on the remote node because it would be getting "500 Internal server errors" and retrying to connect to the reporting server several times before failing.

@rottenbytes

This comment has been minimized.

Show comment
Hide comment
@rottenbytes

rottenbytes Jul 28, 2015

Note for people who would stumble on this : this bug also hits when you make nginx listen on a particular IP

bookshelf["vip"] = "a.b.c.d"

will allow to change the IP that erchef will use to contact bookshelf

rottenbytes commented Jul 28, 2015

Note for people who would stumble on this : this bug also hits when you make nginx listen on a particular IP

bookshelf["vip"] = "a.b.c.d"

will allow to change the IP that erchef will use to contact bookshelf

@diegows

This comment has been minimized.

Show comment
Hide comment
@diegows

diegows Aug 6, 2015

Same problem here, I have to revert back non_ssl_port to 80 to upload cookbooks.

diegows commented Aug 6, 2015

Same problem here, I have to revert back non_ssl_port to 80 to upload cookbooks.

@jessehu

This comment has been minimized.

Show comment
Hide comment
@jessehu

jessehu Aug 14, 2015

I also met this problem in open source chef server 12.1.2.

In my /etc/opscode/chef-server.rb :
nginx['ssl_port'] = 9443
nginx['non_ssl_port'] = 9080

After add the ssl port number into s3_url in /var/opt/opscode/opscode-erchef/sys.config and run 'sudo chef-server-ctl restart', the cookbooks are uploaded:

              {s3_url, "https://hostname:9443"},

jessehu commented Aug 14, 2015

I also met this problem in open source chef server 12.1.2.

In my /etc/opscode/chef-server.rb :
nginx['ssl_port'] = 9443
nginx['non_ssl_port'] = 9080

After add the ssl port number into s3_url in /var/opt/opscode/opscode-erchef/sys.config and run 'sudo chef-server-ctl restart', the cookbooks are uploaded:

              {s3_url, "https://hostname:9443"},
@liku

This comment has been minimized.

Show comment
Hide comment
@liku

liku Sep 12, 2015

+1 faced same issue.

liku commented Sep 12, 2015

+1 faced same issue.

@bluejaguar

This comment has been minimized.

Show comment
Hide comment
@bluejaguar

bluejaguar Sep 17, 2015

+1 , bumping as appears to be same issue we are hitting. chef-server-core-12.1.2-1.el5.x86_64.rpm
I had to hack /var/opt/opscode/opscode-erchef/sys.config to insert the port number into the end of {s3_url,"https://machinename"},
As this issue was raised back on Dec 23, 2014 can we please get this fixed before Dec 23, 2015?

bluejaguar commented Sep 17, 2015

+1 , bumping as appears to be same issue we are hitting. chef-server-core-12.1.2-1.el5.x86_64.rpm
I had to hack /var/opt/opscode/opscode-erchef/sys.config to insert the port number into the end of {s3_url,"https://machinename"},
As this issue was raised back on Dec 23, 2014 can we please get this fixed before Dec 23, 2015?

@jstange

This comment has been minimized.

Show comment
Hide comment
@jstange

jstange Oct 23, 2015

Can corroborate here with chef-server-core-12.2.0-1.el6.x86_64.

jstange commented Oct 23, 2015

Can corroborate here with chef-server-core-12.2.0-1.el6.x86_64.

@kkeane

This comment has been minimized.

Show comment
Hide comment
@kkeane

kkeane Nov 20, 2015

Still happening with chef-server-core-12.3.0-1.el6

kkeane commented Nov 20, 2015

Still happening with chef-server-core-12.3.0-1.el6

@marcparadise

This comment has been minimized.

Show comment
Hide comment
@marcparadise

marcparadise Nov 20, 2015

Member

This tests out locally via both manual cookbook upload tests and pedant run: #637

Member

marcparadise commented Nov 20, 2015

This tests out locally via both manual cookbook upload tests and pedant run: #637

@wduncanfraser

This comment has been minimized.

Show comment
Hide comment
@wduncanfraser

wduncanfraser Feb 14, 2016

Still seeing this with Chef server 12.4.1.

As workaround after reading the above comments, changed line 220 in /opt/opscode/embedded/cookbooks/private-chef/templates/default/oc_erchef.config.erb from

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

to

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>:<%= node['private_chef']['nginx']['ssl_port'] %>"},

Additionally, to configure reporting properly, I changed line 36 in /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/pedant_config.rb.erb from

chef_server "https://<%= node['private_chef']['lb']['api_fqdn'] %>"

to

chef_server "https://<%= node['private_chef']['lb']['api_fqdn'] %>:<%= node['private_chef']['nginx']['ssl_port'] %>"

and line 91 in /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/oc_reporting.config.erb from

{chef_service, [{root_url, "https://<%= node['private_chef']['lb']['vip'] %>"},

to

{chef_service, [{root_url, "https://<%= node['private_chef']['lb']['vip'] %>:<%= node['private_chef']['nginx']['ssl_port'] %>"},

wduncanfraser commented Feb 14, 2016

Still seeing this with Chef server 12.4.1.

As workaround after reading the above comments, changed line 220 in /opt/opscode/embedded/cookbooks/private-chef/templates/default/oc_erchef.config.erb from

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>"},

to

{s3_url, "<%= node['private_chef']['nginx']['x_forwarded_proto'] %>://<%= @helper.vip_for_uri('bookshelf') %>:<%= node['private_chef']['nginx']['ssl_port'] %>"},

Additionally, to configure reporting properly, I changed line 36 in /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/pedant_config.rb.erb from

chef_server "https://<%= node['private_chef']['lb']['api_fqdn'] %>"

to

chef_server "https://<%= node['private_chef']['lb']['api_fqdn'] %>:<%= node['private_chef']['nginx']['ssl_port'] %>"

and line 91 in /opt/opscode-reporting/embedded/cookbooks/opscode-reporting/templates/default/oc_reporting.config.erb from

{chef_service, [{root_url, "https://<%= node['private_chef']['lb']['vip'] %>"},

to

{chef_service, [{root_url, "https://<%= node['private_chef']['lb']['vip'] %>:<%= node['private_chef']['nginx']['ssl_port'] %>"},
@zbuttram

This comment has been minimized.

Show comment
Hide comment
@zbuttram

zbuttram Apr 19, 2016

Just a note for others that might end up here, I had this problem but the fixes weren't fixing the issue completely. If you end up there, make sure your chef-server can resolve its own FQDN as well, that was the extra roadblock in my case.

zbuttram commented Apr 19, 2016

Just a note for others that might end up here, I had this problem but the fixes weren't fixing the issue completely. If you end up there, make sure your chef-server can resolve its own FQDN as well, that was the extra roadblock in my case.

@marcparadise

This comment has been minimized.

Show comment
Hide comment
@marcparadise

marcparadise May 20, 2016

Member

I believe #833 has resolved this oleg-z, would you be able to confirm?

Member

marcparadise commented May 20, 2016

I believe #833 has resolved this oleg-z, would you be able to confirm?

@kkeane

This comment has been minimized.

Show comment
Hide comment
@kkeane

kkeane Jul 4, 2016

Is this pull request in chef-server-core 12.7.0? The problem still exists in that version.

I do notice the changes in oc_erchef.config.erb

I suspect the problem now is that the attribute default['private_chef']['bookshelf']['vip_port'] defaults to 443 and as far as I can tell is never set to match the custom nginx port.

kkeane commented Jul 4, 2016

Is this pull request in chef-server-core 12.7.0? The problem still exists in that version.

I do notice the changes in oc_erchef.config.erb

I suspect the problem now is that the attribute default['private_chef']['bookshelf']['vip_port'] defaults to 443 and as far as I can tell is never set to match the custom nginx port.

@zooda56

This comment has been minimized.

Show comment
Hide comment
@zooda56

zooda56 Aug 18, 2016

This is absolutely correct. The attribute default['private_chef']['bookshelf']['vip_port'] defaults to 443.
Defaults can be overwritten, so in order to keep chef on non-standard post, two attributes must be configured in /etc/opscode/chef-server.rb

nginx['ssl_port'] = 4000
bookshelf['vip_port'] = 4000

Everything runs fine and cookbooks can be uploaded.

zooda56 commented Aug 18, 2016

This is absolutely correct. The attribute default['private_chef']['bookshelf']['vip_port'] defaults to 443.
Defaults can be overwritten, so in order to keep chef on non-standard post, two attributes must be configured in /etc/opscode/chef-server.rb

nginx['ssl_port'] = 4000
bookshelf['vip_port'] = 4000

Everything runs fine and cookbooks can be uploaded.

jstange added a commit to cloudamatic/mu that referenced this issue Aug 18, 2016

@dixon1234

This comment has been minimized.

Show comment
Hide comment
@dixon1234

dixon1234 Jan 24, 2017

Not able to login if set custom nginx ssl port number
nginx['ssl_port']=8444
bookshelf['vip_port'] = 8444

When i login in chef UI i get this error

500Smell something burning?Sorry we have a small fire in the kitchen.An unexpected error has occurred. Our staff are manning the fire extinguishers and have been alerted.

dixon1234 commented Jan 24, 2017

Not able to login if set custom nginx ssl port number
nginx['ssl_port']=8444
bookshelf['vip_port'] = 8444

When i login in chef UI i get this error

500Smell something burning?Sorry we have a small fire in the kitchen.An unexpected error has occurred. Our staff are manning the fire extinguishers and have been alerted.

@ChadScott

This comment has been minimized.

Show comment
Hide comment
@ChadScott

ChadScott Feb 2, 2018

Just wasted hours on this. Can someone please add the port into the s3_url line in the template?

ChadScott commented Feb 2, 2018

Just wasted hours on this. Can someone please add the port into the s3_url line in the template?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment