CSD Upgrade Process

Periodically CSDs will need to be updated to account for new features in the underlying service or to fix bugs. If services don't exist on any cluster with the CSD service type, then an administrator can just replace the CSD in the repository and restart Cloudera Manager.

Special consideration is required if there is a service on any of the clusters that is an instance of the CSD service type. In this case, outright replacing the CSD jar on the filesystem might have some adverse affects depending on the change. In most cases, when the CSD is upgraded (replaced on disk with a new version) Cloudera Manager will try and upgrade the service instance to work with the latest CSD. The administrator is then required to restart the service and redeploy client configuration (if it has any).

We classify the types of changes into: breaking and non-breaking. It is the responsibility of the CSD author to indicate if the new CSD is incompatible (has breaking changes) with previous CSDs. This is done through the use of the CSD generations. Each service that gets created gets the generation of the CSD installed at the time. If Cloudera Manager has services that have been created with an earlier generation number, the new CSD with the incremented generation number cannot be installed. The administrator will need to perform a hard upgrade of the CSD. This is the most intrusive way of performing an upgrade as the service will incur downtime and custom configurations will be lost.

How do you know if you need to increment the generation?

When there is a breaking change to a new CSD, the developer needs to increment the generation number. Most changes to the CSD will not require a generation increment - thus being non-breaking changes. Below is a breakdown of breaking and non-breaking changes.

Non-breaking Changes

Changes to the following fields/structures are non-breaking but might require some intervention from the administrator. A restart of the related services will still be required.

Change	Procedure in Cloudera Manager
label, pluralLabel or description	CM will update the text.
runAs	Next time the service starts, it will run as the new user/group.
version	None; this is opaque to CM so it will just be updated internally.
icon	CM will update the icon.
service maxInstances	If this changes to a value less than the number of running services, there will be a validation error.
parcel	New repository is added and new tags are evaluated on restart.
inExpressWizard	Service will begin appearing as an option in wizards.
rolesWithExternalLinks	Once the service restarts, the new links will appear.
Removing an item from serviceDependencies	CM will stop surfacing the dependency and eventually it will be deleted from the database.
Adding an item to serviceDependencies	If more than one instance of the service type exists, CM will surface a configuration validation error to notify the user to select the service instance.
serviceInit	Existing services won't be affected. New services once added in the wizard will have the new commands executed.
Adding/Removing commands	Commands will show up or disappear for role. This includes service commands, role commands, hdfsDirs.
Adding a new parameter	CM will add the new parameter to the role or service. If the parameter is required, the default value will be used if specified; otherwise there will be a validation error in the configuration.
Removing an existing parameter	CM will not surface the parameter and eventually it will be deleted from the database.
Adding a new role	If the new role is required (minInstances > 0) there will be a validation error on the service page. Otherwise the user can add the new role at anytime.
Removing an existing role	The users will need to be instructed to stop all roles of this type and remove them.
Any parameter default value	CM will record the new default and use it if the user didn't set their own.
parameter configName	The configuration key will change accordingly.
parameter type	CM will try to coerce whatever configuration is in the database into the new type that is in the new CSD. If we can't a configuration validation error will be surfaced.
Any Script Runner	CM will just call the new script with the new arguments and environment variables.
Adding/Removing configWriter	CM will generate/not-generate the appropriate configuration.
Role maxInstances or minInstances	After the change, if minInstances < # running instances < maxInstances, user will not be prompted. Otherwise, role assignments will be shown and user would need to pick roles to add/remove.
External links in roles	When role restarts, new links will appear.
logging	When role restarts, new log directory will be created.
control scripts	These get redeployed when the service restarts.

Breaking Changes

All the changes that do not fall in the non-breaking changes list are considered breaking. The CSD author needs to be aware of this and increment the CSD generation. This will require the user to hard upgrade. Common breaking changes occur anytime an entity identifier changes in the SDL. This includes renaming:

• service name
• CM will see the CSD as a new serviceType.
• role name
• CM will see it as one role being removed and a new role existing. If the service has a role of this type instantiated, the CM will be in an erroneous state after the upgrade.
• parameters
• Since the name is used as the identifier, when the new CSD is installed, it will look like a removal and an addition of the parameter.
• compatibility
• All compatibility changes are breaking by definition.

Hard Upgrade

A hard CSD upgrade is required when the new CSD is of a new generation. The administrator will need to perform the following steps:

1. Stop all services of the CSD service type.
2. Remove the services from Cloudera Manager.
3. This will remove all the service configuration.
4. Update the CSD jar file on disk.
5. Re-add the service and any custom configuration it had before.

Additional documentation from the CSD developer might be required for this type of upgrade.