Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't hardcode the registry URL to authenticate to, use the API response #1525

Merged
merged 2 commits into from
Feb 6, 2024

Conversation

ashb
Copy link
Contributor

@ashb ashb commented Feb 2, 2024

Description

There were a few paths where the registry to log in to was hardcoded
based on the environment (but not all of them -- for instance the Push
path using the go client library was already using this field to auth against!)

This change makes the Push and Pull flow use the pre-existing
ImageRepository flow in the Deploy(ment) response from Astro.

This also removes a bit of duplicated code -- various places were
stripping of the Bearer prefix from the password before calling the
Push function, but that function itself did that.

🧪 Functional Testing

I have used this to deploy to Astro cloud without trouble

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

@ashb
Copy link
Contributor Author

ashb commented Feb 2, 2024

I haven't yet tested this against the Houston API. That needs to happen.

Copy link

codecov bot commented Feb 2, 2024

Codecov Report

Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (22ca8f6) 85.96% compared to head (0d6ec95) 86.01%.

❗ Current head 0d6ec95 differs from pull request most recent head 377aa0e. Consider uploading reports for the commit 377aa0e to get more accurate results

Files Patch % Lines
airflow/docker_image.go 67.56% 7 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1525      +/-   ##
==========================================
+ Coverage   85.96%   86.01%   +0.05%     
==========================================
  Files         112      112              
  Lines       14962    14961       -1     
==========================================
+ Hits        12862    12869       +7     
+ Misses       1265     1261       -4     
+ Partials      835      831       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kushalmalani
Copy link
Contributor

I haven't yet tested this against the Houston API. That needs to happen.

Adding @rujhan-arora-astronomer @rishkarajgi to confirm the software/deploy changes

@kushalmalani
Copy link
Contributor

Ran e2e tests from a binary of your branch, and tests around deploy look good - https://app.circleci.com/pipelines/github/astronomer/astro/115485/workflows/8cfe0817-29a4-4e92-b81b-c05534b7c3bc/jobs/5304060

Copy link
Contributor

@vandyliu vandyliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code changes look good
tests have good coverage
i'm not too familiar with the docker pull/push part of this code so hopefully someone with more context can also approve

Copy link
Contributor

@kushalmalani kushalmalani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. If possible, can you add some more tests on uncovered lines to bump the coverage

@ashb ashb force-pushed the dont-hardcode-registry-url branch 3 times, most recently from 214e260 to 9ba49a9 Compare February 6, 2024 13:15
@ashb ashb force-pushed the dont-hardcode-registry-url branch from 9ba49a9 to 9cff3a1 Compare February 6, 2024 13:23
@ashb
Copy link
Contributor Author

ashb commented Feb 6, 2024

I've increased coverage a little bit and refactored to get it a bit higher, but hitting the patch target isn't practical I don't think

There were a few paths where the registry to log in to was hardcoded
based on the environment (but not all of them -- for instance the Push
path using the go client library was already using this field to auth against!)

This change makes the Push and Pull flow use the pre-existing
`ImageRepository` flow in the Deploy(ment) response from Astro.

This also removes a bit of duplicated code -- various places were
stripping of the `Bearer ` prefix from the password before calling the
Push function, but that function itself did that.
@ashb ashb force-pushed the dont-hardcode-registry-url branch from 9cff3a1 to 0d6ec95 Compare February 6, 2024 14:44
Copy link
Contributor

@rujhan-arora-astronomer rujhan-arora-astronomer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except uncovered lines

Copy link
Contributor

@neel-astro neel-astro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a minor nit and a scenario to validate, but otherwise LGTM, a good cleanup 🧹

airflow/docker_image.go Outdated Show resolved Hide resolved
}
return nil
parts := strings.SplitN(imageName, "/", 2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we try this change out with a custom registry on Software, not sure if it will break or not @rujhan-arora-astronomer @rishkarajgi
with a custom registry, our image name would be of the form fmt.Sprintf("%s:%s", registry, fmt.Sprintf("%s-%s", name, tag))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, there's a good chance that this won't work in that case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have the domain here, so we could look if we are in custom registry mode and return.... registry? in that case?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I recollect correctly, the reason we had such a structure was that some of the custom registry endpoints were of the form: custom-registry.com/dev-airflow, and it won't allow us to append /<image_name> after that.

The only part I am uncertain about is whether trying to log in to just custom-registry.com would work or not.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait a sec, for Software we don't pass username, so we won't try to login during image push, so I think we are good 🙂.
For software either we log in to the registry during astro login or we expect the customer to log in manually for the custom registry.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah!

The login has to be to custom-registry.com (i.e. without the repo, but it's not always a bare hostname, it could be internal.company.com/registry for instance)

So we're good to merge this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup

Co-authored-by: Neel Dalsania <neel.dalsania@astronomer.io>
@ashb ashb merged commit 5b83c93 into main Feb 6, 2024
3 checks passed
@ashb ashb deleted the dont-hardcode-registry-url branch February 6, 2024 18:21
kushalmalani pushed a commit that referenced this pull request Feb 7, 2024
…nse (#1525)

There were a few paths where the registry to log in to was hardcoded
based on the environment (but not all of them -- for instance the Push
path using the go client library was already using this field to auth against!)

This change makes the Push and Pull flow use the pre-existing
`ImageRepository` flow in the Deploy(ment) response from Astro.

This also removes a bit of duplicated code -- various places were
stripping of the `Bearer ` prefix from the password before calling the
Push function, but that function itself did that.

Co-authored-by: Neel Dalsania <neel.dalsania@astronomer.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants