Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: helper script to read KeyVault secrets #1859

Merged
merged 11 commits into from Feb 8, 2024
Merged

Conversation

thekaveman
Copy link
Member

@thekaveman thekaveman commented Jan 26, 2024

Part of #1847

What this PR does

Todo

Done

The existing Terraform app_service definition includes a system-defined managed identity.

Further, this managed identity should be able to GET secrets via an existing policy definition.

According to the Python package docs, DefaultCredential will check for access via this system-defined managed identity when running in Azure. This PR adds a DEBUG-only route /testsecret to help verify access from within Azure.

How to verify the POC

  1. Update your local .env file to add a new variable

     testsecret=Hello from the local environment!
    
  2. Rebuild and Reopen the devcontainer

  3. Call the helper script, setting the DJANGO_SETTINGS_MODULE env var and passing the name of the secret to read

     DJANGO_SETTINGS_MODULE=benefits.settings \
     python benefits/secrets.py testsecret
    
  4. Verify output:

     [local] testsecret: Hello from the local environment!
    
  5. From the devcontainer terminal, login to the Azure CLI with the --use-device-code flag, following instructions printed in the terminal. You may need to login to Azure on the website first, if you receive an error related to 2FA, to trigger that flow. Then run:

     az login --use-device-code
    
  6. Call the helper script again configuring DJANGO_ALLOWED_HOSTS for the dev environment:

     DJANGO_ALLOWED_HOSTS=dev-benefits.calitp.org \
     DJANGO_SETTINGS_MODULE=benefits.settings \
     python benefits/secrets.py testsecret
    
  7. Verify output:

     [dev] testsecret: Hello from the dev KeyVault!
    
  8. Launch the local app with F5 and visit the /testsecret endpoint, verify output in the browser:

     Hello from the local environment!
    
  9. Override DJANGO_ALLOWED_HOSTS in .vscode/launch.json to point to dev. Be sure to include localhost so you can launch the app!

     "env": {
       ...
       "DJANGO_ALLOWED_HOSTS": "dev-benefits.calitp.org,localhost",
     }
    
  10. Launch the local app with F5 and visit the /testsecret endpoint, verify output in the browser:

    Hello from the dev KeyVault!
    
  11. Try another call against e.g. the test environment:

     DJANGO_ALLOWED_HOSTS=test-benefits.calitp.org \
     DJANGO_SETTINGS_MODULE=benefits.settings \
     python benefits/secrets.py <another secret name>
    
  12. And verify the expected output:

     [test] <secret name>: <secret value from test>
    

Post-merge verification steps

After this PR is merged and deployed to dev, visit the /testsecret route, which should print the value of the test secret in the browser (shown in localhost below):

image

Confirmed in dev

image

benefits/secrets.py Fixed Show resolved Hide resolved
@github-actions github-actions bot added back-end Django views, sessions, middleware, models, migrations etc. deployment-dev [auto] Changes that will trigger a deploy if merged to dev labels Jan 26, 2024
Copy link

github-actions bot commented Jan 26, 2024

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  benefits
  secrets.py 53-62
  sentry.py 88
  settings.py
  urls.py 42-47
Project Total  

This report was generated by python-coverage-comment-action

benefits/secrets.py Fixed Show fixed Hide fixed
benefits/secrets.py Fixed Show fixed Hide fixed
benefits/settings.py Fixed Show fixed Hide fixed
benefits/settings.py Dismissed Show dismissed Hide dismissed
benefits/secrets.py Fixed Show fixed Hide fixed
function rather than variable to enable dynamic runtime calculation
e.g. for unit tests
move module-level variables into configure function, unneeded outside
module, wait for Django startup before using settings
similar to how this is done in the Terraform module
@thekaveman thekaveman force-pushed the feat/secrets-helper branch 2 times, most recently from 1371de3 to c542647 Compare January 30, 2024 22:34
Copy link

benefits/secrets.py Fixed Show fixed Hide fixed
@cal-itp cal-itp deleted a comment from github-actions bot Jan 31, 2024
@cal-itp cal-itp deleted a comment from github-actions bot Jan 31, 2024
@cal-itp cal-itp deleted a comment from github-actions bot Jan 31, 2024
@cal-itp cal-itp deleted a comment from github-actions bot Jan 31, 2024
allow interfacing with Azure inside devcontainer
the secret does not contain any sensitive information
and is only configured in the dev environment
Copy link

github-actions bot commented Feb 2, 2024

@thekaveman thekaveman marked this pull request as ready for review February 2, 2024 00:21
@thekaveman thekaveman requested a review from a team as a code owner February 2, 2024 00:21
@thekaveman thekaveman self-assigned this Feb 2, 2024
@angela-tran
Copy link
Member

Verifying the POC

I was able to go through steps 1-4 and 6 without any issue. ✅

For step 5, I was able to go to /testsecret locally and see the secret in my browser.

Just curious though: do you see these errors in the terminal in VS Code?

[02/Feb/2024 21:05:33] DEBUG benefits.core.session:251 Update session debug
[02/Feb/2024 21:05:33] INFO azure.identity._credentials.environment:109 No environment configuration found.
[02/Feb/2024 21:05:33] INFO azure.identity._credentials.managed_identity:96 ManagedIdentityCredential will use IMDS
[02/Feb/2024 21:05:33] DEBUG azure.identity._internal.decorators:53 EnvironmentCredential.get_token failed: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/decorators.py", line 33, in wrapper
    token = fn(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/environment.py", line 150, in get_token
    raise CredentialUnavailableError(message=message)
azure.identity._exceptions.CredentialUnavailableError: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
[02/Feb/2024 21:05:33] INFO azure.core.pipeline.policies.http_logging_policy:514 Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED'
Request method: 'GET'
Request headers:
    'User-Agent': 'azsdk-python-identity/1.15.0 Python/3.11.6 (Linux-6.5.11-linuxkit-x86_64-with-glibc2.36)'
No body was attached to the request
[02/Feb/2024 21:05:33] DEBUG urllib3.connectionpool:244 Starting new HTTP connection (1): 169.254.169.254:80
[02/Feb/2024 21:05:33] DEBUG urllib3.connectionpool:546 http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net HTTP/1.1" 403 None
[02/Feb/2024 21:05:33] INFO azure.core.pipeline.policies.http_logging_policy:550 Response status: 403
Response headers:
    'Connection': 'close'
[02/Feb/2024 21:05:33] DEBUG charset_normalizer:439 Encoding detection: ascii is most likely the one.
[02/Feb/2024 21:05:33] DEBUG azure.identity._internal.get_token_mixin:101 ImdsCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/core/pipeline/policies/_universal.py", line 614, in deserialize_from_text
    return json.loads(data_as_str)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/managed_identity_client.py", line 41, in _process_response
    content = ContentDecodePolicy.deserialize_from_text(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/core/pipeline/policies/_universal.py", line 616, in deserialize_from_text
    raise DecodeError(
azure.core.exceptions.DecodeError: JSON is invalid: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/imds.py", line 87, in _request_token
    self._client.request_token(*scopes, connection_timeout=1, retry_total=0)
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/managed_identity_client.py", line 113, in request_token
    token = self._process_response(response, request_time)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/managed_identity_client.py", line 45, in _process_response
    if response.http_response.content_type.startswith("application/json"):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/get_token_mixin.py", line 86, in get_token
    token = self._request_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/imds.py", line 97, in _request_token
    raise CredentialUnavailableError(error_message) from ex
azure.identity._exceptions.CredentialUnavailableError: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
[02/Feb/2024 21:05:33] DEBUG azure.identity._internal.decorators:53 ManagedIdentityCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/core/pipeline/policies/_universal.py", line 614, in deserialize_from_text
    return json.loads(data_as_str)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/managed_identity_client.py", line 41, in _process_response
    content = ContentDecodePolicy.deserialize_from_text(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/core/pipeline/policies/_universal.py", line 616, in deserialize_from_text
    raise DecodeError(
azure.core.exceptions.DecodeError: JSON is invalid: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/imds.py", line 87, in _request_token
    self._client.request_token(*scopes, connection_timeout=1, retry_total=0)
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/managed_identity_client.py", line 113, in request_token
    token = self._process_response(response, request_time)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/managed_identity_client.py", line 45, in _process_response
    if response.http_response.content_type.startswith("application/json"):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/decorators.py", line 33, in wrapper
    token = fn(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/managed_identity.py", line 137, in get_token
    return self._credential.get_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/get_token_mixin.py", line 86, in get_token
    token = self._request_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/imds.py", line 97, in _request_token
    raise CredentialUnavailableError(error_message) from ex
azure.identity._exceptions.CredentialUnavailableError: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
[02/Feb/2024 21:05:33] DEBUG azure.identity._persistent_cache:105 msal-extensions is unable to encrypt a persistent cache: "Unable to import module 'gi'
Runtime dependency of PyGObject is missing.
Depends on your Linux distro, you could install it system-wide by something like:
    sudo apt install python3-gi python3-gi-cairo gir1.2-secret-1
If necessary, please refer to PyGObject's doc:
https://pygobject.readthedocs.io/en/latest/getting_started.html
"
Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/msal_extensions/libsecret.py", line 18, in <module>
    import gi  # https://github.com/AzureAD/microsoft-authentication-extensions-for-python/wiki/Encryption-on-Linux  # pylint: disable=line-too-long
    ^^^^^^^^^
ModuleNotFoundError: No module named 'gi'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_persistent_cache.py", line 101, in _get_persistence
    return msal_extensions.LibsecretPersistence(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/msal_extensions/persistence.py", line 314, in __init__
    from .libsecret import (  # This uncertain import is deferred till runtime
  File "/home/calitp/.local/lib/python3.11/site-packages/msal_extensions/libsecret.py", line 20, in <module>
    raise ImportError("""Unable to import module 'gi'
ImportError: Unable to import module 'gi'
Runtime dependency of PyGObject is missing.
Depends on your Linux distro, you could install it system-wide by something like:
    sudo apt install python3-gi python3-gi-cairo gir1.2-secret-1
If necessary, please refer to PyGObject's doc:
https://pygobject.readthedocs.io/en/latest/getting_started.html

[02/Feb/2024 21:05:33] DEBUG azure.identity._internal.decorators:53 SharedTokenCacheCredential.get_token failed: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
Traceback (most recent call last):
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/decorators.py", line 33, in wrapper
    token = fn(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/shared_cache.py", line 80, in get_token
    return self._credential.get_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_credentials/shared_cache.py", line 124, in get_token
    account = self._get_account(self._username, self._tenant_id, is_cae=is_cae)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/decorators.py", line 79, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/calitp/.local/lib/python3.11/site-packages/azure/identity/_internal/shared_token_cache.py", line 205, in _get_account
    raise CredentialUnavailableError(message=NO_ACCOUNTS)
azure.identity._exceptions.CredentialUnavailableError: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
[02/Feb/2024 21:05:33] DEBUG azure.identity._internal.decorators:34 AzureCliCredential.get_token succeeded
[02/Feb/2024 21:05:33] DEBUG azure.identity._internal.decorators:48 [Authenticated account] Client ID: <ID>. Tenant ID: <ID>. User Principal Name: unavailableUpn. Object ID (user): <ID>
[02/Feb/2024 21:05:33] INFO azure.identity._credentials.chained:95 DefaultAzureCredential acquired a token from AzureCliCredential
[02/Feb/2024 21:05:33] INFO azure.core.pipeline.policies.http_logging_policy:514 Request URL: 'https://<KEY VAULT URL>/secrets/testsecret/?api-version=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': 'ce000502-c20e-11ee-b4dc-0242ac150004'
    'User-Agent': 'azsdk-python-keyvault-secrets/4.7.0 Python/3.11.6 (Linux-6.5.11-linuxkit-x86_64-with-glibc2.36)'
    'Authorization': 'REDACTED'
No body was attached to the request

@thekaveman
Copy link
Member Author

@angela-tran

Just curious though: do you see these errors in the terminal in VS Code?

Yes! I hadn't noticed these before, but I do see them in trying to hit the local /testsecret endpoint.

I think this reflects azure-identity's stated strategy for how DefaultAzureCredential attempts to resolve:

DefaultAzureCredential attempts to authenticate via the following mechanisms, in this order, stopping when one succeeds...
...
As of version 1.14.0, DefaultAzureCredential will attempt to authenticate with all developer credentials until one succeeds, regardless of any errors previous developer credentials experienced. For example, a developer credential may attempt to get a token and fail, so DefaultAzureCredential will continue to the next credential in the flow.

Since these are DEBUG level messages, and because it follows the published docs, I'm not too worried about it.

angela-tran
angela-tran previously approved these changes Feb 5, 2024
@thekaveman
Copy link
Member Author

@machikoyasuda please make sure you can go through the How to Verify steps above before we merge this.

@thekaveman thekaveman mentioned this pull request Feb 6, 2024
4 tasks
@thekaveman thekaveman marked this pull request as draft February 6, 2024 23:53
@thekaveman
Copy link
Member Author

Moving back to draft as I work on some additional fixes for running the app locally and unit/integration testing.

used to shortcut secret store for e.g. local testing
Copy link

github-actions bot commented Feb 7, 2024

benefits/settings.py Dismissed Show dismissed Hide dismissed
benefits/settings.py Dismissed Show dismissed Hide dismissed
benefits/secrets.py Fixed Show fixed Hide fixed
@thekaveman thekaveman marked this pull request as ready for review February 7, 2024 03:46
Copy link

github-actions bot commented Feb 7, 2024

@thekaveman
Copy link
Member Author

@angela-tran @machikoyasuda ready for re-review! Please see the updated steps in How to verify the POC above.

Copy link
Member

@angela-tran angela-tran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to confirm support for local configuration

Copy link
Member

@machikoyasuda machikoyasuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Able to go thru all the testing steps and read KV secret values for local, dev and test

@thekaveman thekaveman merged commit 9e310a7 into dev Feb 8, 2024
15 checks passed
@thekaveman thekaveman deleted the feat/secrets-helper branch February 8, 2024 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
back-end Django views, sessions, middleware, models, migrations etc. deployment-dev [auto] Changes that will trigger a deploy if merged to dev
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

3 participants