Api endpoint to get deployment manifest with expanded runtime config

[gberche-orange]

Is your feature request related to a problem? Please describe.

As a bosh operator, in order to record a reliable current state of a bosh deployment, I need a deployment manifest with the applied runtime config. Then I archive this deployment manifest (typically in git) in order to track history of changes applied by my automation pipelines.

Currently, the runtime config may change, but the current endpoint has not yet options to render the applied runtime config. For cloud-config, it does support indicating the cloud-config within the deployment is outdated, see

bosh/src/bosh-director/spec/unit/api/controllers/deployments_controller_spec.rb

Lines 1073 to 1100 in 69dea96

	it 'mark cloud-config outdated if it references a deleted config' do
	deleted_cloud_config = Models::Config.make(:cloud, raw_manifest: {}, deleted: true)
	Models::Deployment.create(
	name: 'deployment-4',
	).tap do |deployment|
	deployment.cloud_configs = [deleted_cloud_config]
	end
	get '/', {}, {}
	expect(last_response.status).to eq(200)
	body = JSON.parse(last_response.body)
	expect(body).to eq(
	[
	{
	'name' => 'deployment-1',
	'releases' => [
	{ 'name' => 'release-1', 'version' => '1' },
	{ 'name' => 'release-1', 'version' => '2' },
	],
	'stemcells' => [
	{ 'name' => 'stemcell-1', 'version' => '1' },
	{ 'name' => 'stemcell-2', 'version' => '1' },
	],
	'cloud_config' => 'outdated',
	'teams' => %w[dabest daworst],
	'locked' => false,
	},

Describe the solution you'd like

A new param to the /deployments/<deployment> endpoint to render the deployment manifest along with the add-ons.

Describe alternatives you've considered

I looked for existing task debug traces and could not find a dumped rendered manifest.

Looking at source code, the planner below seems to perform the manifest merge but does not yet dump it

bosh/src/bosh-director/lib/bosh/director/deployment_plan/planner_factory.rb

Lines 50 to 126 in 4f06568

	def parse_from_manifest(manifest, cloud_config_consolidator, runtime_configs, options)
	runtime_config_consolidator = Bosh::Director::RuntimeConfig::RuntimeConfigsConsolidator.new(runtime_configs)
	@manifest_validator.validate(manifest.manifest_hash)
	cloud_manifest = manifest.cloud_config_hash
	manifest.resolve_aliases
	manifest_hash = manifest.manifest_hash
	@logger.debug("Deployment manifest:\n#{manifest_hash}")
	@logger.debug("Cloud config manifest:\n#{cloud_manifest}")
	name = manifest_hash['name']
	deployment_model = @deployment_repo.find_or_create_by_name(name, options)
	deployment_model.add_variable_set(created_at: Time.now, writable: true) if deployment_model.variable_sets.empty?
	plan_options = {
	'is_deploy_action' => !!options['deploy'],
	'recreate' => !!options['recreate'],
	'recreate_persistent_disks' => options['recreate_persistent_disks'] == true,
	'fix' => !!options['fix'],
	'skip_drain' => options['skip_drain'],
	'job_states' => options['job_states'] || {},
	'max_in_flight' => validate_and_get_argument(options['max_in_flight'], 'max_in_flight'),
	'canaries' => validate_and_get_argument(options['canaries'], 'canaries'),
	'tags' => parse_tags(manifest_hash, runtime_config_consolidator),
	}
	@logger.info('Creating deployment plan')
	@logger.info("Deployment plan options: #{plan_options}")
	deployment = Planner.new(
	name,
	manifest.manifest_hash,
	manifest.manifest_text,
	cloud_config_consolidator.cloud_configs,
	[],
	deployment_model,
	plan_options,
	manifest_hash.fetch('properties', {}),
	)
	deployment.cloud_planner = CloudManifestParser.new(@logger).parse(cloud_manifest)
	DeploymentSpecParser.new(deployment, Config.event_log, @logger).parse(manifest_hash, plan_options)
	unless deployment.addons.empty?
	deployment.addons.each do |addon|
	addon.add_to_deployment(deployment)
	end
	end
	variables_spec_parser = Bosh::Director::DeploymentPlan::VariablesSpecParser.new(@logger, deployment.model)
	variables_interpolator = Bosh::Director::ConfigServer::VariablesInterpolator.new
	runtime_configs.each do |runtime_config|
	parsed_runtime_config = RuntimeConfig::RuntimeManifestParser.new(@logger, variables_spec_parser).parse(variables_interpolator.interpolate_runtime_manifest(runtime_config.raw_manifest, name))
	runtime_config_applies = false
	runtime_config_applies = true if parsed_runtime_config.addons.empty?
	parsed_runtime_config.get_applicable_releases(deployment).each do |release|
	runtime_config_applies = true
	release.add_to_deployment(deployment)
	end
	parsed_runtime_config.addons.each do |addon|
	runtime_config_applies = true
	addon.add_to_deployment(deployment)
	end
	deployment.add_variables(parsed_runtime_config.variables)
	if runtime_config_applies
	deployment.runtime_configs.push(runtime_config)
	end
	end
	DeploymentValidator.new.validate(deployment)
	deployment
	end

A clear and concise description of what you want to happen.

I'm also considering fetching the full manifest directly from the director db, but would need guidance into the current director db schema, as https://github.com/cloudfoundry/bosh/tree/main/docs/director_schema seems outdated

[jpalermo]

The /deployments/:deployment/diff endpoint does use runtime configs to generate the diff between the old manifest and the new manifest.

bosh/src/bosh-director/lib/bosh/director/api/controllers/deployments_controller.rb

Lines 469 to 489 in 6b40dfc

	if deployment
	before_manifest = Manifest.load_from_model(deployment, resolve_interpolation: false)
	before_manifest.resolve_aliases
	teams = deployment.teams
	else
	before_manifest = Manifest.generate_empty_manifest
	teams = Bosh::Director::Models::Team.transform_admin_team_scope_to_teams(token_scopes)
	end
	after_cloud_configs = Bosh::Director::Models::Config.latest_set_for_teams('cloud', *teams)
	after_runtime_configs = Bosh::Director::Models::Config.latest_set_for_teams('runtime', *teams)
	after_manifest = Manifest.load_from_hash(
	manifest_hash,
	manifest_text,
	after_cloud_configs,
	after_runtime_configs,
	resolve_interpolation:
	false,
	)
	after_manifest.resolve_aliases

Seems like a lot of that code could be used to do what you want on /deployments/:deployment.

Since that endpoint currently renders the "currently deployed manifest", if you're adding an option to add runtime configs, it feels correct that the option should have a flag for "currently deployed manifest + runtime configs", but also adding a flag for "currently deployed manifest + new runtime configs" seems like a fine option too.

You might also be able to just use the diff endpoint to achieve what you want without any changes, but I'll leave that up to you.

[gberche-orange]

Thanks @jpalermo for your detailed analysis and suggestion for contribution !