Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support gcp secret manager #11436

Open
wants to merge 26 commits into
base: master
Choose a base branch
from

Conversation

HuanXin-Chen
Copy link
Contributor

Description

Many enterprises are utilizing cloud services from AWS and GCP, relying on the secret manager provided by these platforms to handle sensitive information. Integrating Apache APISIX with these secret managers can streamline the process of using sensitive information within APISIX, enabling users to manage and utilize cloud-stored sensitive data more conveniently, thus enhancing the overall security and usability of the system.

This PR has completed the support for GCP. It added the gcp.lua file to the original secret module, allowing users to store their secret information on GCP using the same reference method as before.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

apisix/secret/gcp.lua Outdated Show resolved Hide resolved
@dosubot dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Aug 9, 2024
@bzp2010 bzp2010 self-requested a review September 4, 2024 07:40
apisix/utils/google-cloud-oauth.lua Outdated Show resolved Hide resolved
apisix/utils/google-cloud-oauth.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
t/secret/conf/success.json Show resolved Hide resolved
@bzp2010 bzp2010 requested a review from kayx23 September 4, 2024 08:02
@bzp2010
Copy link
Contributor

bzp2010 commented Sep 4, 2024

Hi @kayx23

Please take a look at the documentation when you are free to see if there are any problems. 🫡

apisix/utils/google-cloud-oauth.lua Outdated Show resolved Hide resolved
apisix/utils/google-cloud-oauth.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/utils/google-cloud-oauth.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Show resolved Hide resolved
apisix/secret/gcp.lua Show resolved Hide resolved
t/secret/gcp.t Outdated Show resolved Hide resolved
- secret_name: the secret name in the secrets management service
- key: get the value of a property when the value of the secret is a JSON string

### Required Parameters
Copy link
Member

@kayx23 kayx23 Sep 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This portion of the doc perhaps should also be added to Admin API doc given the current structure of the apache/apisix doc since that is where the vault parameters are also documented: https://apisix.apache.org/docs/apisix/admin-api/#request-body-parameters-11

cc @pottekkat for inputs as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I can do it in the new PR, because I didn't add the AWS part before.
#11417

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That works as well :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I have completed:#11569

t/secret/gcp.t Outdated

__DATA__

=== TEST 1: sanity
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change sanity to a meaningful title, and it should contain schema and validate

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All Resolved.
I have fixed these issues, please review them again.

apisix/secret/gcp.lua Outdated Show resolved Hide resolved
Comment on lines +181 to +184
local res, err = make_request_to_gcp(conf, main_key)
if not res then
return nil, "failed to retrtive data from gcp secret manager: " .. err
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can remove make_

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is consistent with any of the remaining implementations, both hc vault and aws. aws follows the style of vault, so I think it's fine to keep it the same.

If you think it's critical, you can naturally change all three secret providers at once after that PR merge, but unfortunately I don't see the need to do that necessarily.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I also think there's no need to modify it.


local data, err = core.json.decode(res)
if not data then
return nil, "failed to decode result, res: " .. res .. ", err: " .. err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to remove res, it is dangerous to contain res.body in error message

aud = self.token_uri,
scope = self.scope,
iat = get_timestamp(),
exp = get_timestamp() + (60 * 60)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

60 * 60, it is a magic number, pls add a comment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we know exactly what it means because the code is a simple copy of the apisix/plugins/google-cloud-logging/oauth.lua file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to create an issue to make apisix/plugins/google-cloud-logging/oauth.lua use this new file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, when this PR is merged, I will create a new PR to use this new file.

@HuanXin-Chen
Copy link
Contributor Author

@bzp2010 @kayx23 @membphis @soulbird I have fixed some issues in the code. Please review it again when you have time. Thank you very much. 🙏

@bzp2010
Copy link
Contributor

bzp2010 commented Sep 12, 2024

@HuanXin-Chen There were some failures in CI involving traffic-split3.t, no worries, just merge the master branch into your dev branch feat-gcp-secret, we've fixed that.

apisix/secret/gcp.lua Show resolved Hide resolved
apisix/secret/gcp.lua Outdated Show resolved Hide resolved
apisix/secret/gcp.lua Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants