Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

osd: allow for injecting extra env. variables via ConfigMap #9647

Merged
merged 1 commit into from
Jan 26, 2022
Merged

osd: allow for injecting extra env. variables via ConfigMap #9647

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 19 additions & 0 deletions deploy/examples/osd-env-override.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# ###############################################################################################################
# The `rook-ceph-osd-env-override` ConfigMap is a development feature
# that allows to inject arbitrary environment variables to OSD-related
# containers created by the operator.
# ###############################################################################################################

apiVersion: v1
kind: ConfigMap
metadata:
name: rook-ceph-osd-env-override
namespace: rook-ceph
data:
# Bypass the ASan's assertion that it is the very first loaded DSO.
# This is necessary for crimson-osd as it's currently built with
# the ASan sanitizer turned on which means the `libasan.so` must
# the be the very first loaded dynamic library. Unfortunately, this
# isn't fulfilled as the containers use `ld.preload`, so ASan was
# aborting the entire OSD.
ASAN_OPTIONS: verify_asan_link_order=0
14 changes: 14 additions & 0 deletions pkg/operator/ceph/cluster/osd/envs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ const (
osdWalSizeEnvVarName = "ROOK_OSD_WAL_SIZE"
osdsPerDeviceEnvVarName = "ROOK_OSDS_PER_DEVICE"
osdDeviceClassEnvVarName = "ROOK_OSD_DEVICE_CLASS"
osdConfigMapOverrideName = "rook-ceph-osd-env-override"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this more generic and apply it to all the daemons instead of just the OSD?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just requested the opposite here. :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@travisn why not? This might be useful in the future for other daemons too..., looks simple too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about if we merge this and follow up on this discussion? It will help unblock @rzarzynski.

For the follow-up, this is more like a development-only feature. This change is needed to work with Crimson in the short term, but is there another scenario? For any scenario we are aware of, the operator is expected to add env vars on the pod spec instead of the admin customizing it, so I'm not sure it's worth doing this for other daemons.

// EncryptedDeviceEnvVarName is used in the pod spec to indicate whether the OSD is encrypted or not
EncryptedDeviceEnvVarName = "ROOK_ENCRYPTED_DEVICE"
PVCNameEnvVarName = "ROOK_PVC_NAME"
Expand Down Expand Up @@ -213,6 +214,19 @@ func osdActivateEnvVar() []v1.EnvVar {
return append(cephVolumeEnvVar(), monEnvVars...)
}

func getEnvFromSources() []v1.EnvFromSource {
optionalConfigMapRef := true

return []v1.EnvFromSource{
{
ConfigMapRef: &v1.ConfigMapEnvSource{
travisn marked this conversation as resolved.
Show resolved Hide resolved
LocalObjectReference: v1.LocalObjectReference{Name: osdConfigMapOverrideName},
Optional: &optionalConfigMapRef,
},
},
}
}

func getTcmallocMaxTotalThreadCacheBytes(tcmallocMaxTotalThreadCacheBytes string) v1.EnvVar {
var value string
// If empty we read the default value from the file coming with the package
Expand Down
1 change: 1 addition & 0 deletions pkg/operator/ceph/cluster/osd/provision_spec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,7 @@ func (c *Cluster) provisionOSDContainer(osdProps osdProperties, copyBinariesMoun
Image: c.spec.CephVersion.Image,
VolumeMounts: volumeMounts,
Env: envVars,
EnvFrom: getEnvFromSources(),
SecurityContext: &v1.SecurityContext{
Privileged: &privileged,
RunAsUser: &runAsUser,
Expand Down
3 changes: 3 additions & 0 deletions pkg/operator/ceph/cluster/osd/spec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,7 @@ func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionC
Image: c.rookVersion,
VolumeMounts: configVolumeMounts,
Env: configEnvVars,
EnvFrom: getEnvFromSources(),
SecurityContext: securityContext,
})
}
Expand Down Expand Up @@ -549,6 +550,7 @@ func (c *Cluster) makeDeployment(osdProps osdProperties, osd OSDInfo, provisionC
Image: c.spec.CephVersion.Image,
VolumeMounts: volumeMounts,
Env: envVars,
EnvFrom: getEnvFromSources(),
Resources: osdProps.resources,
SecurityContext: securityContext,
StartupProbe: controller.GenerateStartupProbeExecDaemon(opconfig.OsdType, osdID),
Expand Down Expand Up @@ -767,6 +769,7 @@ func (c *Cluster) getActivateOSDInitContainer(configDir, namespace, osdID string
VolumeMounts: volMounts,
SecurityContext: controller.PrivilegedContext(true),
Env: envVars,
EnvFrom: getEnvFromSources(),
Resources: osdProps.resources,
}

Expand Down