Skip to content

Commit

Permalink
- Create baton migration to migrate any helm repository URLs if oldHe…
Browse files Browse the repository at this point in the history
…lmRepositoryUrl system property set

- Update baton migration to migrate any old helm module names from extensions-helm-<chart-name> to aissemble-<chart-name>-chart
- Refactored existing helm chart name migrations
  • Loading branch information
chang-annie committed May 17, 2024
1 parent e7b1a40 commit 362db56
Show file tree
Hide file tree
Showing 15 changed files with 478 additions and 81 deletions.
30 changes: 16 additions & 14 deletions DRAFT_RELEASE_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The baseline Kafka Docker image has moved from using the _wurstmeister/kafka_ im
* Python modules were renamed to reflect aiSSEMBLE. These include the following.

| Old Python Module | New Python Module |
| --------------------------------------- |---------------------------------------------------|
|-----------------------------------------|---------------------------------------------------|
| foundation-core-python | aissemble-core-python |
| foundation-model-training-api | aissemble-foundation-model-training-api |
| foundation-versioning-service | aissemble-foundation-versioning-service |
Expand All @@ -26,7 +26,7 @@ The baseline Kafka Docker image has moved from using the _wurstmeister/kafka_ im
| foundation-transform-core-python | aissemble-Foundation-transform-core-python |
| extensions-model-training-api-sagemaker | aissemble-extensions-model-training-api-sagemaker |
| extensions-data-delivery-spark-py | aissemble-extensions-data-delivery-spark-py |
| extensions-encryption-valut-python | aissemble-extensions-encryption-vault-python |
| extensions-encryption-vault-python | aissemble-extensions-encryption-vault-python |
| extensions-transform-spark-python | aissemble-extensions-transform-spark-python |
| test-data-delivery-pyspark-model | aissemble-test-data-delivery-pyspark-model |
| test-data-delivery-pyspark-model-basic | aissemble-test-data-delivery-pyspark-model-basic |
Expand Down Expand Up @@ -89,18 +89,20 @@ The following steps will upgrade your project to 1.7. These instructions consist
## Automatic Upgrades
To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the `build-parent` version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

| Migration Name | Description |
|-------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| upgrade-tiltfile-aissemble-version-migration | Updates the aiSSEMBLE version within your project's Tiltfile |
| upgrade-v2-chart-files-aissemble-version-migration | Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-mlflow-v2-external-s3-migration | Update the mlflow V2 deployment (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments |
| upgrade-spark-application-s3-migration | Update the pipeline SparkApplication(s) (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments |
| upgrade-foundation-extension-python-package-migration | Updates the pyproject.toml files within your projects pipelines folder (<YOUR_PROJECT>-pipelines) to use the updated aiSSEMBLE foundation and extension Python packages with the latest naming convention |
| upgrade-helm-chart-files-names-migration | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm chart naming convention (aissemble-\<chart-name\>-chart) |
| upgrade-dockerfile-pip-install-migration | Updates dockerfiles such that python dependency installations fail during the build, rather than at runtime |
| enable-habushu-build-cache-migration | Updates the `pom.xml` file for any Habushu-managed modules to ensure that the build directory is specified. |
| data-lineage-package-import-migration | Updates the package imports for all java files that are referencing `com.boozallen.aissemble.data.lineage`.
| Migration Name | Description |
|-------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| upgrade-tiltfile-aissemble-version-migration | Updates the aiSSEMBLE version within your project's Tiltfile |
| upgrade-v2-chart-files-aissemble-version-migration | Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-mlflow-v2-external-s3-migration | Update the mlflow V2 deployment (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments |
| upgrade-spark-application-s3-migration | Update the pipeline SparkApplication(s) (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments |
| upgrade-foundation-extension-python-package-migration | Updates the pyproject.toml files within your projects pipelines folder (<YOUR_PROJECT>-pipelines) to use the updated aiSSEMBLE foundation and extension Python packages with the latest naming convention |
| upgrade-helm-chart-names-migration | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm chart naming convention (`aissemble-<chart-name>-chart`). |
| upgrade-helm-module-names-migration | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm module naming convention (`aissemble-<chart-name>-chart`) |
| upgrade-helm-chart-repository-url-migration | Updates the Helm repository URL within your project's deploy Chart.yaml file to point to ghcr.io. Only runs if the previous Helm chart repository URL is passed in through the `oldHelmRepositoryUrl` system property (using `-DoldHelmRepositoryUrl`) |
| upgrade-dockerfile-pip-install-migration | Updates dockerfiles such that python dependency installations fail during the build, rather than at runtime |
| enable-habushu-build-cache-migration | Updates the `pom.xml` file for any Habushu-managed modules to ensure that the build directory is specified. |
| data-lineage-package-import-migration | Updates the package imports for all java files that are referencing `com.boozallen.aissemble.data.lineage`. |

To deactivate any of these migrations, add the following configuration to the `baton-maven-plugin` within your root `pom.xml`:

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
package com.boozallen.aissemble.upgrade.migration;

/*-
* #%L
* aiSSEMBLE::Foundation::Upgrade
* %%
* Copyright (C) 2021 Booz Allen
* %%
* This software package is licensed under the Booz Allen Public License. All Rights Reserved.
* #L%
*/

import com.boozallen.aissemble.upgrade.util.FileUtils;
import org.apache.commons.collections4.CollectionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.technologybrewery.baton.BatonException;

import java.io.File;
import java.io.IOException;
import java.util.List;

import static com.boozallen.aissemble.upgrade.util.FileUtils.getRegExCaptureGroups;

/**
* Baton migration used to migrate the Helm Chart names and/or Helm module names in the Chart.yaml and values.yaml files within a project's deploy folder
*/
public abstract class AbstractHelmNameMigration extends AbstractAissembleMigration{
private static final Logger logger = LoggerFactory.getLogger(AbstractHelmNameMigration.class);

@Override
protected boolean shouldExecuteOnFile(File file) {
boolean shouldExecute = false;
try {
List<String> namesToUpdate = getRegExCaptureGroups(getReplacementRegex(file), file);
shouldExecute = CollectionUtils.isNotEmpty(namesToUpdate);
} catch (IOException e){
throw new BatonException("Unable to determine if Helm chart migration must be executed", e);
}
return shouldExecute;
}

@Override
protected boolean performMigration(File file) {
boolean performedSuccessfully = false;
try {
performedSuccessfully = FileUtils.replaceInFile(
file,
getReplacementRegex(file),
getReplacementText(file)
);
} catch (Exception e) {
logger.error("Unable to perform Helm chart migration due to exception", e);
}
return performedSuccessfully;
}

public abstract String getReplacementRegex(File file);
public abstract String getReplacementText(File file);

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
package com.boozallen.aissemble.upgrade.migration;

/*-
* #%L
* aiSSEMBLE::Foundation::Upgrade
* %%
* Copyright (C) 2021 Booz Allen
* %%
* This software package is licensed under the Booz Allen Public License. All Rights Reserved.
* #L%
*/

import com.boozallen.aissemble.upgrade.util.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;

/**
* Baton migration used to migrate the Helm Chart repository URL in the Chart.yaml within a project's deploy folder.
* Migration is triggered by setting the oldHelmRepositoryUrl system property.
*/
public abstract class AbstractHelmRepositoryMigration extends AbstractAissembleMigration{
public static final Logger logger = LoggerFactory.getLogger(AbstractHelmRepositoryMigration.class);
private static final String OLD_HELM_REPO_PROPERTY = "oldHelmRepositoryUrl";

protected String getOldHelmRepositoryUrl(){
String oldHelmRepositoryUrl = System.getProperty(OLD_HELM_REPO_PROPERTY);
if (oldHelmRepositoryUrl!= null){
return oldHelmRepositoryUrl;
} else {
logger.info("oldHelmRepositoryUrl system property not set. Skipping helm chart repository URL migration.");
return null;
}
}

/**
* Function that checks if user has provided system property containing old Helm repository URL
* If yes, then Helm Charts repository URL migration will be run.
* Otherwise, migration is skipped.
*/
@Override
protected boolean shouldExecuteOnFile(File file) {
boolean shouldExecute = false;
if (getOldHelmRepositoryUrl() != null){
shouldExecute = true;
}
return shouldExecute;
}

@Override
protected boolean performMigration(File file) {
boolean performedSuccessfully = false;
try {
performedSuccessfully = FileUtils.replaceInFile(
file,
getReplacementRegex(),
getReplacementText()
);
} catch (Exception e) {
logger.error("Unable to perform helm chart repository URL migration due to exception", e);
}
return performedSuccessfully;
}

public abstract String getReplacementRegex();
public abstract String getReplacementText();
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
package com.boozallen.aissemble.upgrade.migration.v1_7_0;

/*-
* #%L
* aiSSEMBLE::Foundation::Upgrade
* %%
* Copyright (C) 2021 Booz Allen
* %%
* This software package is licensed under the Booz Allen Public License. All Rights Reserved.
* #L%
*/

import com.boozallen.aissemble.upgrade.migration.AbstractHelmNameMigration;

import java.io.File;

/**
* Baton migration used to migrate the Chart.yaml and values*.yaml files within a project's deploy folder
* to use new Helm module names
* Example: extensions-helm-kafka to aissemble-kafka-chart
*/
public class HelmChartsModuleMigration extends AbstractHelmNameMigration {
private static final String moduleNameReplaceRegex = "extensions-helm-([^\\/\\n:]+)";
private static final String AISSEMBLE_PREFIX = "aissemble-";
private static final String CHART_POSTFIX = "-chart";

@Override
public String getReplacementRegex(File file) {
return moduleNameReplaceRegex;
}

@Override
public String getReplacementText(File file) {
return AISSEMBLE_PREFIX.concat(FIRST_REGEX_GROUPING + CHART_POSTFIX);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -9,90 +9,50 @@
* This software package is licensed under the Booz Allen Public License. All Rights Reserved.
* #L%
*/

import com.boozallen.aissemble.upgrade.migration.AbstractAissembleMigration;
import com.boozallen.aissemble.upgrade.util.FileUtils;
import org.apache.commons.collections4.CollectionUtils;
import com.boozallen.aissemble.upgrade.migration.AbstractHelmNameMigration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.technologybrewery.baton.BatonException;

import java.io.File;
import java.io.IOException;
import java.util.List;
import static com.boozallen.aissemble.upgrade.util.FileUtils.getRegExCaptureGroups;

/**
* Baton migration used to migrate the Chart.yaml and values*.yaml files within a project's deploy folder
* to use the new Helm chart naming convention (aissemble-<chart-name>-chart)
* Example: aissemble-kafka to aissemble-kafka-chart
*/

public class HelmChartsNameMigration extends AbstractAissembleMigration {
public class HelmChartsNameMigration extends AbstractHelmNameMigration {
public static final Logger logger = LoggerFactory.getLogger(HelmChartsNameMigration.class);
private static final String dependencyNameReplaceRegex = "(aissemble-[^\\/\\n:]+)(?<!-chart)$";
private static final String appNameReplaceRegex = "(^\\s*)(aissemble-[^\\/\\n:]+)(?<!-chart):$";
private static final String COLON = ":";
private static final String SECOND_REGEX_GROUPING = "$2";
private static final String CHART_POSTFIX = "-chart";

/**
* Function to check whether migration is necessary for the given file.
* @param file file to check
* @return shouldExecute - whether the migration is necessary.
*/
@Override
protected boolean shouldExecuteOnFile(File file) {
boolean shouldExecute = false;
try {
if (file.getName().matches("Chart.yaml")){
List<String> dependencyNames = getRegExCaptureGroups(dependencyNameReplaceRegex, file);
shouldExecute = CollectionUtils.isNotEmpty(dependencyNames);
} else {
List<String> appNames = getRegExCaptureGroups(appNameReplaceRegex, file);
shouldExecute = CollectionUtils.isNotEmpty(appNames);
}
} catch (IOException e) {
throw new BatonException("Unable to determine if Helm charts should be migrated!", e);
}
public String getDependencyNameReplacementText(){
return FIRST_REGEX_GROUPING.concat(CHART_POSTFIX);
}

if (shouldExecute) {
logger.info("Found Helm chart file using old naming convention. Migrating file: {}", file.getName());
}
return shouldExecute;
public String getAppNameReplacementText(){
return FIRST_REGEX_GROUPING.concat(SECOND_REGEX_GROUPING + CHART_POSTFIX + COLON);
}

/**
* Performs the migration if the shouldExecuteOnFile() returns true.
* @param file file to migrate
* @return performedSuccessfully - Whether the file was migrated successfully.
*/
@Override
protected boolean performMigration(File file) {
boolean performedSuccessfully = false;

try {

String appNameReplacementText = FIRST_REGEX_GROUPING.concat(SECOND_REGEX_GROUPING + CHART_POSTFIX + COLON);
String dependencyNameReplacementText = FIRST_REGEX_GROUPING.concat(CHART_POSTFIX);

if (file.getName().matches("Chart.yaml")){
performedSuccessfully = FileUtils.replaceInFile(
file,
dependencyNameReplaceRegex,
dependencyNameReplacementText
);
} else {
performedSuccessfully = FileUtils.replaceInFile(
file,
appNameReplaceRegex,
appNameReplacementText
);
}
} catch (Exception e) {
logger.error("Unable to perform helm chart name migration due to exception", e);
public String getReplacementRegex(File file) {
if (file.getName().matches("Chart.yaml")){
return dependencyNameReplaceRegex;
} else {
return appNameReplaceRegex;
}

return performedSuccessfully;
}

@Override
public String getReplacementText(File file){
if (file.getName().matches("Chart.yaml")){
return getDependencyNameReplacementText();
} else {
return getAppNameReplacementText();
}
}
}


Loading

0 comments on commit 362db56

Please sign in to comment.