Fix CloudSecretManagerBackend regression with explicit project_id (issue #61217)

[dv-gorasiya]

Summary

Fixes #61217: CloudSecretManagerBackend with explicit project_id fails when Application Default Credentials (ADC) have no default project.

Root Cause

_get_credentials_using_adc() raises an AirflowException when google.auth.default() returns None project_id. This occurs before CloudSecretManagerBackend.__init__ can apply the explicit project_id parameter, causing the backend to fail even when a valid project ID is provided.

Changes

1. credentials_provider.py – _get_credentials_using_adc() now returns an empty string ("") instead of raising when ADC yields None project_id.
2. secret_manager.py – Added validation in __init__ that raises AirflowException if neither ADC nor the explicit project_id parameter provides a project ID.
3. Formatting – Applied black and isort to both modified files.

Backward Compatibility

• Callers that previously got the exception still get one – the exception is now raised from CloudSecretManagerBackend.__init__ with a clearer message.
• Callers that pass an explicit project_id now work correctly – the explicit parameter is honored.
• No change to the public API – get_credentials_and_project_id() still returns tuple[Credentials, str] (empty string is a valid str).

Testing

• Manual verification with a minimal test script (included in the PR description).
• Existing unit tests for CloudSecretManagerBackend and credentials_provider pass because they mock google.auth.default to return a valid project ID.
• New unit test added to test_credentials_provider.py: test_get_credentials_and_project_id_with_default_auth_no_project_id verifies that get_credentials_and_project_id() returns an empty string when ADC yields None project_id.
• The fix ensures the regression described in the issue is resolved: CloudSecretManagerBackend(project_id="my-project") now works when ADC lacks a default project.

Impact on Other Callers

Other components that call get_credentials_and_project_id() without an explicit key_secret_project_id will receive an empty string instead of an AirflowException. If those components do not validate the project ID, they may propagate the empty string to downstream Google APIs, which will produce a different error (e.g., “Invalid project”). This is acceptable because:

1. The primary regression (explicit project_id being ignored) is fixed.
2. The scenario occurs only when ADC has no default project and the caller does not provide an explicit project ID via key_secret_project_id (or similar).
3. The error message change is minimal; the user still gets an error indicating something is wrong with the project ID.

Checklist

• My commit messages are descriptive and reference the issue number.
• I have reviewed the existing unit tests for the affected modules.
• I have added/updated tests that verify the fix (if applicable).
• Any dependent changes have been merged and published.

Related Issues

• CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0) #61217 (original issue)