Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ossm): adds OSSM annotations to the relevant cluster resources #1088

Merged
merged 20 commits into from
Oct 4, 2023
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
bfa5075
feat(ossm): adds OSSM annotations to the relevant cluster resources
cam-garrison Apr 4, 2023
1408c29
fix: adds mesh annotation to the ns
bartoszmajsak Apr 18, 2023
166e409
fix: use project controller created route for ds project routing (#4)
cam-garrison May 23, 2023
47e43ac
feat: use annotation to fetch host for notebook routing (#5)
cam-garrison Jun 26, 2023
dd22cd7
fix: patch notebook annotations (#6)
cam-garrison Jun 28, 2023
c917d76
Linting fixes
cam-garrison Jun 29, 2023
93aa3ae
add featureFlagEnabled() to backend, use for SM
cam-garrison Aug 2, 2023
a0ea817
Merge branch 'main' into ossm_annotations
bartoszmajsak Aug 23, 2023
abf520d
explicitly set sidecar injection annotation
cam-garrison Aug 24, 2023
5b26e64
change to istio inject label from anno
cam-garrison Aug 28, 2023
c106663
Merge branch 'main' into ossm_annotations
cam-garrison Sep 20, 2023
8376c28
remove namespacekind, use projectkind
cam-garrison Sep 21, 2023
48bfe4e
lint getSMGwHost() changes
cam-garrison Sep 21, 2023
6b12f82
move getDashboardConfig call, remove unnec constant
cam-garrison Sep 21, 2023
ca51ef3
simplify annotations fetch
cam-garrison Sep 22, 2023
74b4c79
pass parsed SM flag instead ofdash config
cam-garrison Sep 22, 2023
b55490f
add sm manifests changes
cam-garrison Sep 25, 2023
2d2244c
remove image patch
cam-garrison Oct 4, 2023
104c88b
no longer poll on dashboardconfig, remove trailing slash
cam-garrison Oct 4, 2023
df9f285
remove unnecessary patches and rename patches in kustomization
cam-garrison Oct 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
24 changes: 21 additions & 3 deletions backend/src/routes/api/namespaces/namespaceUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { PatchUtils, V1SelfSubjectAccessReview } from '@kubernetes/client-node';
import { NamespaceApplicationCase } from './const';
import { K8sStatus, KubeFastifyInstance, OauthFastifyRequest } from '../../../types';
import { createCustomError } from '../../../utils/requestUtils';
import { featureFlagEnabled, getDashboardConfig } from '../../../utils/resourceUtils';
import { isK8sStatus, safeURLPassThrough } from '../k8s/pass-through';

const checkNamespacePermission = (
Expand Down Expand Up @@ -60,13 +61,22 @@ export const applyNamespaceChange = async (
throw createCustomError('Forbidden', "You don't have the access to update the namespace", 403);
}

// calling featureFlagEnabled to set the bool to false if it's set to anything but false ('true', undefined, etc)
const enableServiceMesh = featureFlagEnabled(
getDashboardConfig().spec.dashboardConfig.disableServiceMesh,
);

let labels = {};
let annotations = {};
switch (context) {
case NamespaceApplicationCase.DSG_CREATION:
labels = {
'opendatahub.io/dashboard': 'true',
'modelmesh-enabled': 'true',
};
annotations = {
'opendatahub.io/service-mesh': String(enableServiceMesh),
};
break;
case NamespaceApplicationCase.MODEL_SERVING_PROMOTION:
labels = {
Expand All @@ -78,9 +88,17 @@ export const applyNamespaceChange = async (
}

return fastify.kube.coreV1Api
.patchNamespace(name, { metadata: { labels } }, undefined, undefined, undefined, undefined, {
headers: { 'Content-type': PatchUtils.PATCH_FORMAT_JSON_MERGE_PATCH },
})
.patchNamespace(
name,
{ metadata: { labels, annotations } },
undefined,
undefined,
undefined,
undefined,
{
headers: { 'Content-type': PatchUtils.PATCH_FORMAT_JSON_MERGE_PATCH },
},
)
.then(() => ({ applied: true }))
.catch((e) => {
fastify.log.error(
Expand Down
32 changes: 23 additions & 9 deletions backend/src/routes/api/notebooks/utils.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
import { KubeFastifyInstance, Notebook, NotebookData, Route } from '../../../types';
import { KubeFastifyInstance, Notebook, NotebookData } from '../../../types';
import { PatchUtils, V1ContainerStatus, V1Pod, V1PodList } from '@kubernetes/client-node';
import { createCustomError } from '../../../utils/requestUtils';
import { getUserName } from '../../../utils/userUtils';
import { RecursivePartial } from '../../../typeHelpers';
import { featureFlagEnabled, getDashboardConfig } from '../../../utils/resourceUtils';
import {
createNotebook,
generateNotebookNameFromUsername,
getNamespaces,
getNotebook,
getRoute,
getServiceMeshGwHost,
updateNotebook,
} from '../../../utils/notebookUtils';
import { FastifyRequest } from 'fastify';
Expand All @@ -27,12 +29,24 @@ export const getNotebookStatus = async (
const notebookName = notebook?.metadata.name;
let newNotebook: Notebook;
if (isRunning && !notebook?.metadata.annotations?.['opendatahub.io/link']) {
const route = await getRoute(fastify, namespace, notebookName).catch((e) => {
fastify.log.warn(`Failed getting route ${notebookName}: ${e.message}`);
return undefined;
});
if (route) {
newNotebook = await patchNotebookRoute(fastify, route, namespace, notebookName).catch((e) => {
const enableServiceMesh = featureFlagEnabled(
getDashboardConfig().spec.dashboardConfig.disableServiceMesh,
);
let host: string;
if (enableServiceMesh) {
host = await getServiceMeshGwHost(fastify, namespace).catch((e) => {
fastify.log.warn(`Failed getting route ${notebookName}: ${e.message}`);
return undefined;
});
} else {
const route = await getRoute(fastify, namespace, notebookName).catch((e) => {
fastify.log.warn(`Failed getting route ${notebookName}: ${e.message}`);
return undefined;
});
host = route?.spec.host;
}
Comment on lines +36 to +47
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we fall back to retrieving the host through the route?
By enabling the flag without service mesh, we'll get an invalid url.

Suggested change
if (enableServiceMesh) {
host = await getServiceMeshGwHost(fastify, namespace).catch((e) => {
fastify.log.warn(`Failed getting route ${notebookName}: ${e.message}`);
return undefined;
});
} else {
const route = await getRoute(fastify, namespace, notebookName).catch((e) => {
fastify.log.warn(`Failed getting route ${notebookName}: ${e.message}`);
return undefined;
});
host = route?.spec.host;
}
if (enableServiceMesh) {
host = await getServiceMeshGwHost(fastify, namespace).catch((e) => {
fastify.log.warn(`Failed getting service mesh route ${notebookName}: ${e.message}`);
return undefined;
});
}
if (!host) {
const route = await getRoute(fastify, namespace, notebookName).catch((e) => {
fastify.log.warn(`Failed getting route ${notebookName}: ${e.message}`);
return undefined;
});
host = route?.spec.host;
}

if (host) {
cam-garrison marked this conversation as resolved.
Show resolved Hide resolved
newNotebook = await patchNotebookRoute(fastify, host, namespace, notebookName).catch((e) => {
cam-garrison marked this conversation as resolved.
Show resolved Hide resolved
fastify.log.warn(`Failed patching route to notebook ${notebookName}: ${e.message}`);
return notebook;
});
Expand Down Expand Up @@ -71,14 +85,14 @@ export const checkPodContainersReady = (pod: V1Pod): boolean => {

export const patchNotebookRoute = async (
fastify: KubeFastifyInstance,
route: Route,
host: string,
namespace: string,
name: string,
): Promise<Notebook> => {
const patch: RecursivePartial<Notebook> = {
metadata: {
annotations: {
'opendatahub.io/link': `https://${route.spec.host}/notebook/${namespace}/${name}`,
'opendatahub.io/link': `https://${host}/notebook/${namespace}/${name}/`,
},
},
};
Expand Down
4 changes: 4 additions & 0 deletions backend/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ export type DashboardConfig = K8sResourceCommon & {
disableModelServing: boolean;
disableProjectSharing: boolean;
disableCustomServingRuntimes: boolean;
disableServiceMesh: boolean;
modelMetricsNamespace: string;
disablePipelines: boolean;
};
Expand Down Expand Up @@ -410,6 +411,9 @@ export type Notebook = K8sResourceCommon & {
'opendatahub.io/link': string; // redirect notebook url
'opendatahub.io/username': string; // the untranslated username behind the notebook

// Openshift Service Mesh specific annotations. They're needed to orchestrate additional resources for nb namespaces.
'opendatahub.io/service-mesh': string;

// TODO: Can we get this from the data in the Notebook??
'notebooks.opendatahub.io/last-image-selection': string; // the last image they selected
'notebooks.opendatahub.io/last-size-selection': string; // the last notebook size they selected
Expand Down
1 change: 1 addition & 0 deletions backend/src/utils/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ export const blankDashboardCR: DashboardConfig = {
disableModelServing: false,
disableProjectSharing: false,
disableCustomServingRuntimes: false,
disableServiceMesh: true,
modelMetricsNamespace: '',
disablePipelines: false,
},
Expand Down
46 changes: 44 additions & 2 deletions backend/src/utils/notebookUtils.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { getDashboardConfig } from './resourceUtils';
import { featureFlagEnabled, getDashboardConfig } from './resourceUtils';
import {
EnvironmentVariable,
ImageInfo,
Expand Down Expand Up @@ -68,6 +68,32 @@ export const getRoute = async (
return kubeResponse.body as Route;
};

export const getServiceMeshGwHost = async (
fastify: KubeFastifyInstance,
namespace: string,
): Promise<string> => {
const kubeResponse = await fastify.kube.coreV1Api.readNamespace(namespace).catch((res) => {
const e = res.response.body;
const error = createCustomError('Error getting Namespace', e.message, e.code);
fastify.log.error(error);
throw error;
});

const annotations = kubeResponse.body.metadata?.annotations;

if (!annotations || !annotations['service-mesh.opendatahub.io/public-gateway-host-external']) {
const error = createCustomError(
'Annotation not found',
`Could not find annotation 'service-mesh.opendatahub.io/public-gateway-host-external' for namespace: ${namespace}`,
404,
);
fastify.log.error(error);
throw error;
}

return annotations['service-mesh.opendatahub.io/public-gateway-host-external'];
};

export const createRBAC = async (
fastify: KubeFastifyInstance,
namespace: string,
Expand Down Expand Up @@ -256,6 +282,10 @@ export const assembleNotebook = async (
},
}));

const serviceMeshEnabled = String(
!featureFlagEnabled(getDashboardConfig().spec?.dashboardConfig?.disableServiceMesh),
);

return {
apiVersion: 'kubeflow.org/v1',
kind: 'Notebook',
Expand All @@ -265,12 +295,15 @@ export const assembleNotebook = async (
'opendatahub.io/odh-managed': 'true',
'opendatahub.io/user': translatedUsername,
'opendatahub.io/dashboard': 'true',
'sidecar.istio.io/inject': String(serviceMeshEnabled),
},
annotations: {
'notebooks.opendatahub.io/oauth-logout-url': `${url}/notebookController/${translatedUsername}/home`,
'notebooks.opendatahub.io/last-size-selection': notebookSize.name,
'notebooks.opendatahub.io/last-image-selection': imageSelection,
'notebooks.opendatahub.io/inject-oauth': String(!serviceMeshEnabled),
'opendatahub.io/username': username,
'opendatahub.io/service-mesh': serviceMeshEnabled,
'kubeflow-resource-stopped': null,
},
name: name,
Expand Down Expand Up @@ -444,7 +477,16 @@ export const createNotebook = async (
notebookAssembled.metadata.annotations = {};
}

notebookAssembled.metadata.annotations['notebooks.opendatahub.io/inject-oauth'] = 'true';
const enableServiceMesh = featureFlagEnabled(
getDashboardConfig().spec.dashboardConfig.disableServiceMesh,
);

notebookAssembled.metadata.annotations['notebooks.opendatahub.io/inject-oauth'] = String(
!enableServiceMesh,
);
notebookAssembled.metadata.annotations['opendatahub.io/service-mesh'] = String(enableServiceMesh);
notebookAssembled.metadata.labels['sidecar.istio.io/inject'] = String(enableServiceMesh);

const notebookContainers = notebookAssembled.spec.template.spec.containers;

if (!notebookContainers[0]) {
Expand Down
3 changes: 3 additions & 0 deletions backend/src/utils/resourceUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -590,6 +590,9 @@ export const getDashboardConfig = (): DashboardConfig => {
return dashboardConfigWatcher.getResources()?.[0];
};

export const featureFlagEnabled = (disabledSettingState?: boolean): boolean =>
disabledSettingState === false;

Comment on lines +593 to +595
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest adding the same comment as we have present in the frontend code.

Suggested change
export const featureFlagEnabled = (disabledSettingState?: boolean): boolean =>
disabledSettingState === false;
/**
* Feature flags are required in the config -- but upgrades can be mixed and omission of the property
* usually ends up being enabled. This will prevent that as a general utility.
*/
export const featureFlagEnabled = (disabledSettingState?: boolean): boolean =>
disabledSettingState === false;

export const updateDashboardConfig = (): Promise<void> => {
return dashboardConfigWatcher.updateResults();
};
Expand Down
1 change: 1 addition & 0 deletions docs/dashboard_config.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ The following are a list of features that are supported, along with there defaul
| disableProjectSharing | false | Disables Project Sharing from Data Science Projects. |
| disableCustomServingRuntimes | false | Disables Custom Serving Runtimes from the Admin Panel. |
| modelMetricsNamespace | false | Enables the namespace in which the Model Serving Metrics' Prometheus Operator is installed. |
| disableServiceMesh | true | Disables use of service mesh for routing and authorization. |

## Defaults

Expand Down
1 change: 1 addition & 0 deletions frontend/src/__mocks__/mockDashboardConfig.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ export const mockDashboardConfig = ({
disableProjects,
disableModelServing,
disableCustomServingRuntimes,
disableServiceMesh: true,
modelMetricsNamespace: 'test-project',
disablePipelines: false,
disableProjectSharing: false,
Expand Down
48 changes: 41 additions & 7 deletions frontend/src/api/k8s/notebooks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,14 @@ import {
} from '~/concepts/pipelines/elyra/utils';
import { createRoleBinding } from '~/api';
import { Volume, VolumeMount } from '~/types';
import { DashboardConfig } from '~/types';
import { featureFlagEnabled } from '~/utilities/utils';
import { assemblePodSpecOptions, getshmVolume, getshmVolumeMount } from './utils';

const assembleNotebook = (
data: StartNotebookData,
username: string,
enableServiceMesh: boolean,
canEnablePipelines?: boolean,
): NotebookKind => {
const {
Expand Down Expand Up @@ -92,14 +95,16 @@ const assembleNotebook = (
'opendatahub.io/odh-managed': 'true',
'opendatahub.io/user': translatedUsername,
[KnownLabels.DASHBOARD_RESOURCE]: 'true',
'sidecar.istio.io/inject': String(enableServiceMesh),
},
annotations: {
'openshift.io/display-name': notebookName.trim(),
'openshift.io/description': description || '',
'notebooks.opendatahub.io/oauth-logout-url': `${origin}/projects/${projectName}?notebookLogout=${notebookId}`,
'notebooks.opendatahub.io/last-size-selection': notebookSize.name,
'notebooks.opendatahub.io/last-image-selection': imageSelection,
'notebooks.opendatahub.io/inject-oauth': 'true',
'notebooks.opendatahub.io/inject-oauth': String(!enableServiceMesh),
'opendatahub.io/service-mesh': String(enableServiceMesh),
'opendatahub.io/username': username,
},
name: notebookId,
Expand Down Expand Up @@ -187,6 +192,24 @@ const getStopPatch = (): Patch => ({
value: getStopPatchDataString(),
});

const getInjectOAuthPatch = (enableServiceMesh: boolean): Patch => ({
op: 'add',
path: '/metadata/annotations/notebooks.opendatahub.io~1inject-oauth',
value: String(!enableServiceMesh),
});

const getProxyInjectPatch = (enableServiceMesh: boolean): Patch => ({
op: 'add',
path: '/metadata/labels/sidecar.istio.io~1inject',
value: String(enableServiceMesh),
});

const getServiceMeshPatch = (enableServiceMesh: boolean): Patch => ({
op: 'add',
path: '/metadata/annotations/opendatahub.io~1service-mesh',
value: String(enableServiceMesh),
});

export const getNotebooks = (namespace: string): Promise<NotebookKind[]> =>
k8sListResource<NotebookKind>({
model: NotebookModel,
Expand All @@ -210,10 +233,19 @@ export const startNotebook = async (
name: string,
namespace: string,
tolerationChanges: TolerationChanges,
dashboardConfig: DashboardConfig,
enablePipelines?: boolean,
): Promise<NotebookKind> => {
const patches: Patch[] = [];
patches.push(startPatch);
const enableServiceMesh = featureFlagEnabled(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tip: If you are using several times the same constant, you can add it into a utils function and reuse it several times.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I understand. Are you suggesting creating a util function that getsAppContext, then calls featureFlagEnabled to parse the service mesh flag (therefore taking no inputs and returning this bool)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm talking about something like this: https://github.com/opendatahub-io/odh-dashboard/blob/main/frontend/src/pages/projects/projectSharing/utils.ts#L7 is not mandatory but it might be nice to have

dashboardConfig.spec.dashboardConfig.disableServiceMesh,
);

const patches: Patch[] = [
startPatch,
getInjectOAuthPatch(enableServiceMesh),
getServiceMeshPatch(enableServiceMesh),
getProxyInjectPatch(enableServiceMesh),
Comment on lines +245 to +247
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These seem all very related, we may want to look at cleaning that up in the future to make it one "set of patches" rather than 3 separate ones. Just for code cleanness.

];

const tolerationPatch = getTolerationPatch(tolerationChanges);
if (tolerationPatch) {
Expand Down Expand Up @@ -242,9 +274,10 @@ export const startNotebook = async (
export const createNotebook = (
data: StartNotebookData,
username: string,
enableServiceMesh: boolean,
canEnablePipelines?: boolean,
): Promise<NotebookKind> => {
const notebook = assembleNotebook(data, username, canEnablePipelines);
const notebook = assembleNotebook(data, username, enableServiceMesh, canEnablePipelines);

const notebookPromise = k8sCreateResource<NotebookKind>({
model: NotebookModel,
Expand All @@ -264,9 +297,10 @@ export const updateNotebook = (
existingNotebook: NotebookKind,
data: StartNotebookData,
username: string,
enableServiceMesh: boolean,
): Promise<NotebookKind> => {
data.notebookId = existingNotebook.metadata.name;
const notebook = assembleNotebook(data, username);
const notebook = assembleNotebook(data, username, enableServiceMesh);

const oldNotebook = structuredClone(existingNotebook);
const container = oldNotebook.spec.template.spec.containers[0];
Expand All @@ -287,9 +321,10 @@ export const updateNotebook = (
export const createNotebookWithoutStarting = (
data: StartNotebookData,
username: string,
enableServiceMesh: boolean,
): Promise<NotebookKind> =>
new Promise((resolve, reject) =>
createNotebook(data, username).then((notebook) =>
createNotebook(data, username, enableServiceMesh).then((notebook) =>
setTimeout(
() =>
stopNotebook(notebook.metadata.name, notebook.metadata.namespace)
Expand All @@ -299,7 +334,6 @@ export const createNotebookWithoutStarting = (
),
),
);

export const deleteNotebook = (notebookName: string, namespace: string): Promise<K8sStatus> =>
k8sDeleteResource<NotebookKind, K8sStatus>({
model: NotebookModel,
Expand Down