Can't create internal route in k8s

[XanderStrike]

Steps:

1. Create an internal domain cf create-shared-domain apps.internal --internal, see success
2. Create a route on that domain cf create-route apps.internal --hostname myapp

Observed:

$ cf create-route apps.internal --hostname myapp
Creating route myapp.apps.internal for org o / space s as admin...
Unexpected Response
Response Code: 500
Request ID:    0fc071d8-18f9-4097-8242-cd82ddd85cda::2c438cbb-f9ac-4c20-984a-4eedc518ce50
Code: 0, Title: , Detail: {
  "errors": [
    {
      "title": "UnknownError",
      "detail": "An unknown error occurred.",
      "code": 10001
    }
  ]
}
FAILED

With the following error in the api server:

Click to expand!

{
  "timestamp": "2020-10-23T17:04:02.322706673Z",
  "message": "Started POST \"/v3/routes\" for user: 5c9e857c-3b63-4c16-8850-1b6425c3fff8, ip: 34.121.231.63 with vcap-request-id: 233082bb-6b33-49f6-8e3e-c3090b85f603::aeb519d0-5baf-483c-9ce0-35751153da1a at 2020-10-23 17:04:02 UTC",
  "log_level": "info",
  "source": "cc.api",
  "data": {
    "request_guid": "233082bb-6b33-49f6-8e3e-c3090b85f603::aeb519d0-5baf-483c-9ce0-35751153da1a"
  },
  "thread_id": 47239097872620,
  "fiber_id": 47239049916160,
  "process_id": 1,
  "file": "/workspace/middleware/request_logs.rb",
  "lineno": 28,
  "method": "call"
}
{
  "timestamp": "2020-10-23T17:04:02.343653789Z",
  "message": "Request failed: 500: {\"errors\"=>[{\"title\"=>\"UnknownError\", \"detail\"=>\"An unknown error occurred.\", \"code\"=>10001, \"test_mode_info\"=>{\"detail\"=>\"IPv4 requires (4) octets.\", \"title\"=>\"CF-ValidationError\", \"backtrace\"=>[\"/layers/paketo-buildpacks_bundle-install/gems/gems/netaddr-2.0.4/lib/util.rb:169:in `parse_IPv4'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/netaddr-2.0.4/lib/ipv4.rb:23:in `parse'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/netaddr-2.0.4/lib/ipv4net.rb:37:in `parse'\", \"/workspace/app/models/runtime/route.rb:236:in `internal_route_vip_range'\", \"/workspace/app/models/runtime/route.rb:230:in `internal_route_vip_range_len'\", \"/workspace/app/models/runtime/route.rb:222:in `before_save'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1759:in `block in _save'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1019:in `around_save'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1757:in `_save'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1469:in `block (2 levels) in save'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1923:in `block in checked_transaction'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/database/transactions.rb:229:in `block in transaction'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/connection_pool/threaded.rb:88:in `hold'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/database/connecting.rb:270:in `synchronize'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/database/transactions.rb:195:in `transaction'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1923:in `checked_transaction'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1469:in `block in save'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1911:in `checked_save_failure'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/model/base.rb:1469:in `save'\", \"/workspace/app/actions/route_create.rb:22:in `block in create'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/database/transactions.rb:258:in `_transaction'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/database/transactions.rb:233:in `block in transaction'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/database/connecting.rb:270:in `synchronize'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/sequel-5.35.0/lib/sequel/database/transactions.rb:195:in `transaction'\", \"/workspace/app/actions/route_create.rb:21:in `create'\", \"/workspace/app/controllers/v3/routes_controller.rb:76:in `create'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:194:in `process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal/rendering.rb:30:in `process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:132:in `run_callbacks'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/abstract_controller/callbacks.rb:41:in `process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal/rescue.rb:22:in `process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `block in instrument'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/activesupport-5.2.4.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/activesupport-5.2.4.3/lib/active_support/notifications.rb:168:in `instrument'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/abstract_controller/base.rb:134:in `process'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionview-5.2.4.3/lib/action_view/rendering.rb:32:in `process'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:191:in `dispatch'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_controller/metal.rb:252:in `dispatch'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `each'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/journey/router.rb:35:in `serve'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/routing/route_set.rb:840:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/activesupport-5.2.4.3/lib/active_support/callbacks.rb:98:in `run_callbacks'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/activesupport-5.2.4.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/actionpack-5.2.4.3/lib/action_dispatch/middleware/executor.rb:14:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'\", \"/workspace/middleware/request_logs.rb:38:in `call'\", \"/workspace/middleware/security_context_setter.rb:19:in `call'\", \"/workspace/middleware/vcap_request_id.rb:15:in `call'\", \"/workspace/middleware/cors.rb:49:in `call_app'\", \"/workspace/middleware/cors.rb:14:in `call'\", \"/workspace/middleware/request_metrics.rb:12:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/thin-1.7.2/lib/thin/connection.rb:50:in `block in process'\", \"/layers/paketo-buildpacks_bundle-install/gems/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `block in spawn_threadpool'\"]}}]}",
  "log_level": "error",
  "source": "cc.api",
  "data": {
    "request_guid": "233082bb-6b33-49f6-8e3e-c3090b85f603::aeb519d0-5baf-483c-9ce0-35751153da1a"
  },
  "thread_id": 47239097872620,
  "fiber_id": 47239049916160,
  "process_id": 1,
  "file": "/workspace/app/controllers/v3/errors_controller.rb",
  "lineno": 11,
  "method": "internal_error"
}

Expected

The Route resource is written to the kubernetes apiserver. The internal machinery needed to correctly process that resource doesn't exist yet, but it'd be nice to be able to create them 😄

Additional info

@tcdowney says its my fault.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175420211

The labels on this github issue will be updated when the story is started.

[tcdowney]

Superficially the error is because we don't configure the internal_route_vip_range property correctly in capi-k8s-release:
https://github.com/cloudfoundry/capi-k8s-release/blob/10aca7a81d08a88df10797608c6a0afd16a10b50/templates/ccng-config.lib.yml#L117

However, since we don't need to reserve and specify a virtual IP for internal routes on Kubernetes, the correct fix is to figure out how to not attempt to reserve one at all. The code pertaining to vips in the route model is what we'd need to take a look at (thank you @XanderStrike for contributing it 🤗 ):

cloud_controller_ng/app/models/runtime/route.rb

Lines 187 to 238 in 8ee3e0f

	def vip
	vip_offset && internal_route_vip_range.nth(vip_offset).to_s
	end
	def wildcard_host?
	host == '*'
	end
	private
	def vip_offset_exceeds_range?
	return false if vip_offset.nil?
	return true if vip_offset <= 0
	vip_offset > internal_route_vip_range_len
	end
	def before_destroy
	destroy_route_bindings
	super
	end
	def find_next_vip_offset
	# This code courtesy of Jeremy Evans as part of discussion on
	# https://groups.google.com/d/msg/sequel-talk/3GJ8_mOgJ9U/roWJ2sWHAwAJ
	# See SQL self-joins for the reasoning behind this
	n = Route.exclude(vip_offset: 1).
	exclude { vip_offset - 1 =~ Route.select(:vip_offset) }.order(:vip_offset).get { vip_offset - 1 } ||
	(return (Route.max(:vip_offset) || 0) + 1)
	Route.where { vip_offset < n }.reverse(:vip_offset).get { vip_offset + 1 } || 1
	end
	def before_save
	if internal? && vip_offset.nil?
	len = internal_route_vip_range_len
	raise OutOfVIPException.new('out of vip_offset slots') if self.class.exclude(vip_offset: nil).count >= len
	self.vip_offset = find_next_vip_offset
	end
	end
	def internal_route_vip_range_len
	internal_route_vip_range.len - 2
	end
	def internal_route_vip_range
	@internal_route_vip_range ||= begin
	internal_route_vip_range = Config.config.get(:internal_route_vip_range)
	NetAddr::IPv4Net.parse(internal_route_vip_range)
	end
	end

[sethboyles]

closing since cf-for-k8s is no longer under active development