Python: Introduce Pydantic settings

[moonbox3]

Motivation and Context

SK Python is tightly coupled to the use of a .env file to read all secrets, keys, endpoints, and more. This doesn't scale well for users who wish to be able to use environment variables with their SK Applications. By introducing Pydantic Settings, it is possible to use both environment variables as well as have a fall-back to a .env file (via a env_file_path parameter), if desired.

By introducing Pydantic Settings, we are removing the requirement to have to create Text/Embedding/Chat completion objects with an api_key or other previously required information (in the case of AzureChatCompletion that means an endpoint, an api_key, a deployment_name, and an api_version). When the AI connector is created, the Pydantic settings are loaded either via env vars or the fall-back .env file, and that means the user can create a chat completion object like:

chat_completion = OpenAIChatCompletion(service_id="test")

or, to optionally override the ai_model_id env var:

chat_completion = OpenAIChatCompletion(service_id="test", ai_model_id="gpt-4-1106")

Note: we have left the ability to specific an api_key/org_id for OpenAIChatCompletion or a deployment_name, endpoint, base_url, and api_version for AzureChatCompletion as before, but if your settings are configured to use env vars/.env file then there is no need to pass this information.

Description

The PR introduces the use of Pydantic settings and removes the use of the python-dotenv library.

• Closes Python: Remove dependencies to .env file #1779
• Updates notebooks, samples, code and tests to remove the explicit config of api_key or other previous .env files values.
• Adds new unit test config using monkeypatch to simulate env variables for testing
• All unit and integration tests passing

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[eavanvalkenburg]

2 big points, I think it is nicer to parent the settings classes in the same folder as the services, for both AI and memory. and I wonder if allowing manually setting the same settings would make sense, and break less?

[markwallace-microsoft]

Python 3.10 Test Coverage Report •

File	Stmts	Miss	Cover	Missing
semantic_kernel/connectors/ai/google_palm/services
gp_chat_completion.py	66	16	76%	10, 135–140, 151–152, 201, 204, 207–208, 216, 220–221, 229
gp_text_completion.py	38	6	84%	10, 57–58, 64–65, 94
gp_text_embedding.py	27	5	81%	10, 47–48, 57–58
semantic_kernel/connectors/ai/open_ai/services
azure_chat_completion.py	76	34	55%	248, 254–255, 264–265, 270–287, 291–298, 310–321
open_ai_text_completion.py	16	1	94%	128
open_ai_text_embedding.py	13	1	92%	83
semantic_kernel/connectors/search_engine
bing_connector.py	35	35	0%	3–71
semantic_kernel/core_plugins/sessions_python_tool
sessions_python_plugin.py	95	9	91%	76–82, 122, 168, 172
TOTAL	5867	1019	83%

Python 3.10 Unit Test Overview

Tests	Skipped	Failures	Errors	Time
1278	1 💤	0 ❌	0 🔥	13.545s ⏱️

[eavanvalkenburg]

Small fix to make (but in multiple places, didn't add comments for all)

[eavanvalkenburg]

small notes, one thing to consider (but can also be done in a subsequent PR) is to make the memory_storage_base more universal (like ai_settings_base or something) and use that throughout

[eavanvalkenburg]

Awesome!