Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate all Deployment Commands to Core #1425

Merged
merged 158 commits into from
Jan 23, 2024
Merged

Migrate all Deployment Commands to Core #1425

merged 158 commits into from
Jan 23, 2024

Conversation

sunkickr
Copy link
Contributor

@sunkickr sunkickr commented Oct 27, 2023

Description

migrate the following commands to Astro CORE API:

  • astro deploy(just deployment selection)
  • astro deployment list
  • astro deployment create
  • astro deployment create --deployment-file
  • astro deployment update
  • astro deployment update --deployment-file
  • astro deployment logs
  • astro deployment variable create
  • astro deployment variable update
  • astro deployment variable list
  • astro deployment worker-queue create
  • astro deployment worker-queue update
  • astro deployment worker-queue delete
  • astro dev upgrade-test(pull deployment image from astro)

🎟 Issue(s)

Related #XXX

🧪 Functional Testing

manually tested all commands with hybrid/hosted/kubernetes/celery Deployments

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

Copy link
Contributor

@vandyliu vandyliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for all the changes david

@@ -185,10 +189,10 @@ func newDeploymentUpdateCmd(out io.Writer) *cobra.Command {
cmd.Flags().IntVarP(&updateSchedulerAU, "scheduler-au", "s", 0, "The Deployment's Scheduler resources in AUs")
cmd.Flags().IntVarP(&updateSchedulerReplicas, "scheduler-replicas", "r", 0, "The number of Scheduler replicas for the Deployment")
cmd.Flags().BoolVarP(&forceUpdate, "force", "f", false, "Force update: Don't prompt a user before Deployment update")
cmd.Flags().StringVarP(&cicdEnforcement, "enforce-cicd", "", "", "When enabled CI/CD Enforcement where deploys to deployment must use an API Key or Token. This essentially forces Deploys to happen through CI/CD. Possible values disable/enable")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a breaking change? Any reason we are changing this?

if organization.IsOrgHosted() {
cmd.Flags().StringVarP(&clusterType, "cluster-type", "", standard, "The Cluster Type to use for the Deployment. Possible values can be standard or dedicated.")
cmd.Flags().StringVarP(&deploymentType, "type", "", standard, "The Type to use for the Deployment. Possible values can be standard or dedicated.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This too...is there a way for us to support both, so we dont break any customer automations that use the old flag?

// update deployment
environmentVariablesObjects, err := client.ModifyDeploymentVariable(variablesCreateInput)
err = Update(currentDeployment.Id, "", "", "", "", "", "", "", "", "", "", "", "", "", 0, 0, []astroplatformcore.WorkerQueueRequest{}, []astroplatformcore.HybridWorkerQueueRequest{}, newEnvironmentVariables, false, coreClient, platformCoreClient)
Copy link
Contributor

@kushalmalani kushalmalani Jan 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in this PR, but we need to improve the Update function with a proper input and validations. This request with so many input arguments looks bad and prone to errors.

@@ -286,13 +287,17 @@ func Deploy(deployInput InputDeploy, client astro.Client, coreClient astrocore.C
}

if !deployInfo.dagDeployEnabled {
fmt.Println("deployment id:")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove these debug prints?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

if err != nil {
if strings.Contains(err.Error(), dagDeployDisabled) {
fmt.Println("deployment id:")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove these debug prints?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

if err != nil {
return err
}

fmt.Println("CI/CD enforcement:")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these debug prints?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

}

func IsDeploymentStandard(deploymentType astroplatformcore.DeploymentType) bool {
return deploymentType == astroplatformcore.DeploymentTypeSTANDARD
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For old deployment templates files, we had a type HOSTED_SHARED. We should also support that, so as to not break those deployment via a file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

if err != nil {
if strings.Contains(err.Error(), dagDeployDisabled) {
fmt.Println(deployInfo.deploymentID)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this

@@ -286,13 +285,15 @@ func Deploy(deployInput InputDeploy, client astro.Client, coreClient astrocore.C
}

if !deployInfo.dagDeployEnabled {
fmt.Println(deployInfo.deploymentID)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this

Copy link
Contributor

@kushalmalani kushalmalani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clean up the debug prints.

Thanks for addressing all other comments. The work on this PR is massive. Great Job 🎉

@sunkickr sunkickr merged commit 250020b into main Jan 23, 2024
3 of 5 checks passed
@sunkickr sunkickr deleted the deployment-list-core branch January 23, 2024 19:01
ashb added a commit that referenced this pull request Feb 20, 2024
After the migration to the REST API in #1425 it wasn't dealing with secret
variables (which don't have a value anymore).

And while I'm there do a bit of tidying up:
- `_, x := range` instead of slice index access
- Remove the use of `environmentVariablesObjects` global variable!
- Refactoring of selecting the variable key to one place only (rather
  than in multiple)
ashb added a commit that referenced this pull request Feb 21, 2024
After the migration to the REST API in #1425 it wasn't dealing with secret
variables (which don't have a value anymore).

And while I'm there do a bit of tidying up:
- `_, x := range` instead of slice index access
- Remove the use of `environmentVariablesObjects` global variable!
- Refactoring of selecting the variable key to one place only (rather
  than in multiple)
ashb added a commit that referenced this pull request Feb 21, 2024
After the migration to the REST API in #1425 it wasn't dealing with secret
variables (which don't have a value anymore).

And while I'm there do a bit of tidying up:
- `_, x := range` instead of slice index access
- Remove the use of `environmentVariablesObjects` global variable!
- Refactoring of selecting the variable key to one place only (rather
  than in multiple)
kushalmalani pushed a commit that referenced this pull request Feb 21, 2024
After the migration to the REST API in #1425 it wasn't dealing with secret
variables (which don't have a value anymore).

And while I'm there do a bit of tidying up:
- `_, x := range` instead of slice index access
- Remove the use of `environmentVariablesObjects` global variable!
- Refactoring of selecting the variable key to one place only (rather
  than in multiple)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants