feat(tests): add initial integration tests to repo

[ammirsm]

📝 Description

• Added new requirements.txt for the development, as we are not using poetry I just added this which has the requirements of the poetry for development and pytest running.
• Add new test folder for testing the integrations.
• Add new integration test which is basically replica of the intro.ipynb but inside the pytest to run it in CI
• Add integration tests to be ran in the CI.

🛠 Type of Change

• New feature (non-breaking change which adds functionality)

🔍 How Has This Been Tested?

• Added new unit/integration tests

✅ Suggested Checklist:

• I have updated the requirements.txt file
• New and existing unit tests pass locally with my changes
• Pre-Commit checks are passing (locally and remotely)
• Title of your PR / MR corresponds to the required format
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I ran ruff check . --fix-only to appease the lint gods
• Commit message follows required format {label}(dspy): {message}

⚠️ Warnings

We need to clean up and move to poetry completely, some places we are using poetry and some places requirements which is making it very hard to maintain dependencies.

[arnavsinghvi11]

is there a way we can check the stdout during compilation and assert the existing printed message? Bootstrapped 4 full traces after 11 examples in round 0.

[ammirsm]

it's a bit dirty to do that but it is doable to add it here.

I would recommend that for this, refactor the teleprompter and assign some values and function for retuning that. checking stdout wouldn't be the best practice here.

---

I suggest after merging this create a new issue for fixing this.

[arnavsinghvi11]

not fully sure but is it needed to specify a python version here? (can we extend this to >=3.9 as in dspy-ai?) and how does this affect users who don't have 3.9 - (saw a followup to this issue here more so for dspy-ai but similar impact).

[ammirsm]

I haven't tested on other pythons, I would say definitely we can add other python versions testing but I suggest defer it after merging my other PR about moving all dependencies to the poetry.

pip requirements doesn't have good caching system in the CI and if we want to add other python version testing in our CI it would consume our CI resources very fast.

[arnavsinghvi11]

Thanks @ammirsm ! this looks great - excited to have these integration tests on board!
left a few general comments to add comprehensiveness.

Regarding the poetry dependencies, is the plan to still support both requirements.txt and poetry dependencies? it was a bit unclear from the comments above. Also, will we remove the extra requirements.txt before merging the PR?

Additionally, what's the best way to test the integration tests in this PR? Would we have to create mock PRs or run GitHub actions locally? would love to have that documented as well as this PR is being reviewed.

Also probably good to squash the commit history here :).

[ammirsm]

Thanks @ammirsm ! this looks great - excited to have these integration tests on board!
left a few general comments to add comprehensiveness.

thanks @arnavsinghvi11 for reviewing the PR. just replied to your comments and added some changes to the PR regarding to them.

Regarding the poetry dependencies, is the plan to still support both requirements.txt and poetry dependencies? it was a bit unclear from the comments above. Also, will we remove the extra requirements.txt before merging the PR?

In regard to poetry dependencies, we plan to eventually move everything to poetry as it's faster in the CI. After this PR is merged and auto deployment is integrated (#919), we'll proceed with the transition to poetry, which has been initiated here: #819

To clarify further:
Currently, we don't have any way to develop using pytest with our original requirements.txt. All dependencies for development and document generation are in poetry. I've added a new requirements-dev.txt to include all existing requirements for development there. This allows us to maintain everything in CI using pip installer for now. Once we've stabilized the requirements, we'll transition everything to poetry.

Additionally, what's the best way to test the integration tests in this PR? Would we have to create mock PRs or run GitHub actions locally? would love to have that documented as well as this PR is being reviewed.

I didn't quite understand this one but will try to add some clarification, lmk if I haven't got your question right.
To run the tests, you simply execute 'pytest tests_integration' locally. When do these tests run? They're executed with each merge and pull request to ensure they don't break our package. Additionally, Hanna will add another workflow that tests the package installed from PyPi, ensuring our package on PyPi is also functioning correctly.

Also probably good to squash the commit history here :).

haha! definitely will squash it for the merge.

[ammirsm]

it's a bit dirty to do that but it is doable to add it here.

I would recommend that for this, refactor the teleprompter and assign some values and function for retuning that. checking stdout wouldn't be the best practice here.

---

I suggest after merging this create a new issue for fixing this.

[ammirsm]

I haven't tested on other pythons, I would say definitely we can add other python versions testing but I suggest defer it after merging my other PR about moving all dependencies to the poetry.

pip requirements doesn't have good caching system in the CI and if we want to add other python version testing in our CI it would consume our CI resources very fast.

[arnavsinghvi11]

Thanks @ammirsm - the comments look good from my end!

added a new requirements-dev.txt to include all existing requirements for development there. This allows us to maintain everything in CI using pip installer for now. Once we've stabilized the requirements, we'll transition everything to poetry.

I see so just to clarify, we have a temporary requirements-dev.txt for now that will help with the impending poetry migration, and once this is done, we will remove that to avoid the duplicate requirements.txt files?

To run the tests, you simply execute 'pytest tests_integration' locally. When do these tests run? They're executed with each merge and pull request to ensure they don't break our package. Additionally, Hanna will add another workflow that tests the package installed from PyPi, ensuring our package on PyPi is also functioning correctly.

yup this is perfect!

[ammirsm]

thanks for the reply @arnavsinghvi11 .

I see so just to clarify, we have a temporary requirements-dev.txt for now that will help with the impending poetry migration, and once this is done, we will remove that to avoid the duplicate requirements.txt files?

yes sir, basically bc we don't want to install dev dependencies during the package installation we put them in the new requirements file and we will remove it when we moved to poetry.