fix(dependencies): move builds and dependencies to poetry

[ammirsm]

📝 Changes Description

This PR contains the following changes:

• migrated setuptools to poetry
• remove poetry main project stuffs and move them to the tool.dependencies
• export the poetry stuffs to requirements to if someone wanted to use it till June
• update PR structure message.

🛠 Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• This change will expire our current cache and need cache update

🔍 How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Added new unit/integration tests
• Updated one notebook (that tests end-to-end)
• I stared at the code and made sure it makes sense

How to test?

• the way that you can test it is to setup an environment and follow the steps for setting up your poetry and use that.

• we should run notebooks and make sure they works.

• another part that should be tested is the build of the package and push it to pypi, which should be done with poetry build

• To publish a Python package to PyPI using Poetry, you need to follow these steps:

1. Poetry will create a distribution package that you can upload to PyPI. You can build the package with the following command:

poetry build

2. Before you can publish the package, you need to add the PyPI repository to Poetry if you haven't done so already. You can do this with the following command:

poetry config repositories.pypi https://upload.pypi.org/legacy/

4. Now you can publish your package to PyPI. You'll be prompted to enter your PyPI username and password:

poetry publish -r pypi

Checklist

• I have added dependencies to the pyproject.toml file
• I have updated the poetry.lock file
• I have updated the requirements.txt file (till 2024-06-01)
• I have updated the documentation accordingly
• New and existing unit tests pass locally with my changes
• New and existing integration notebooks pass locally with my changes
• Pre-Commit checks are passing (locally and remotely)
• Title of your PR / MR corresponds to the required format
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a new notebook that tests the new feature
• I ran ruff check . --fix-only to appease the lint gods
• Commit message follows required format {label}(dspy): {message}

[isaacbmiller]

Thank you for writing this! It is definitely a change that needed to be made at some point. We should keep requirements.txt imo. I tried getting it to autogenerate when I was making CI changes a little while ago, and it was messy, but if you have a better/clean way I am open.

[isaacbmiller]

Why did this get changed?

[ammirsm]

we need to change all of these notebooks to use the poetry, but this one still have some issues with the cache in the local setup which I am trying to solve it with @okhat , would be happy to chat if you can give me a hand on that.

[ammirsm]

@isaacbmiller thank you for the review and comments.

I have added it to recommended Suggested Checklist for templates and I think we should at some point deprecate it.

[isaacbmiller]

Do we have an easy way for people to verify that the notebooks run correctly? Seems like that might be hard to feasibly test. Can we add a CI test for it easily (out of scope for this PR)

[isaacbmiller]

Otherwise just remove the line about notebooks

[ammirsm]

I would say we should make it somehow easy to test and run at least locally and right now I have an issue with running them locally with jupyter and the cache! did you set that up?

And I agree they should be moved to the CI, or I was thinking about move them as integration tests for the system in the pytest but still the same issue with setting up the tests to use the cache which I tried to fix here:
#606

[isaacbmiller]

Also, @okhat should be a blocking reviewer on this one, as he uploads the builds to pypi. I talked to him like a month ago about moving away from setup.py, and I thinking the testing wasn't good enough, but now it is significantly better

[ammirsm]

I agree, but still the PR is a draft till I talk with @okhat

[ammirsm]

I would say the only thing that make me stop is that why locally I can not setup running the notebooks, if you would give me the hand i think it should be gtg.

[ammirsm]

I would say we should make it somehow easy to test and run at least locally and right now I have an issue with running them locally with jupyter and the cache! did you set that up?

And I agree they should be moved to the CI, or I was thinking about move them as integration tests for the system in the pytest but still the same issue with setting up the tests to use the cache which I tried to fix here:
#606

[ammirsm]

@isaacbmiller @okhat folks should I merge the master with this or we should wait on this one for now?

[isaacbmiller]

Overall lgtm but I still think that @okhat should be a blocking reviewer on this

[isaacbmiller]

Why did this file get changed?

[ammirsm]

import should change and we make sure it is working with the poetry import?

[isaacbmiller]

Ah yeah forgot I asked above lol. lgtm