chore: Upgrade deps and pipeline restructuring

[chrisk314]

Summary

This PR updates locked dependencies to latest versions - this is mostly for latest Ray version which needs to be matched with Ray cluster version. Also, restructures the lint-test pipeline to split into parallalel jobs to reduce end-to-end execution time.

Changes

• Upgrades locked dependencies
• Removes version constraint on anyio test dependency
• Splits lint-test pipeline into multiple jobs to reduce end-to-end execution time.

[gemini-code-assist]

Hello @chrisk314, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

This pull request focuses on two main areas: upgrading dependencies and restructuring the lint-test pipeline. The dependency upgrades primarily target the Ray version, ensuring it matches the Ray cluster version. Additionally, the lint-test pipeline has been split into parallel jobs to reduce the overall execution time.

Highlights

• Dependency Upgrades: Locked dependencies have been updated to their latest versions, with a focus on aligning the Ray version with the Ray cluster version. The llama-index dependencies have also been updated.
• Pipeline Restructuring: The lint-test pipeline has been restructured into multiple parallel jobs to decrease the end-to-end execution time.
• Dependency Constraint Removal: The version constraint on the anyio test dependency has been removed.

Changelog

• pyproject.toml
  • Updated locked dependencies to latest versions, including that-depends, llama-index-core, llama-index-llms-openai, and ray[default].
  • Removed version constraint on anyio test dependency, changing anyio>=4.3.0,<4.4.0 to anyio>=4.3.0.
  • Updated llm dependencies from llama-index>=0.12.11 to llama-index-core>=0.12.30 and llama-index-llms-openai>=0.3.33.
  • Moved ray[default]>=2.42.1 from the docs optional dependencies to the main ray optional dependencies.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Dependencies align,
Tests in parallel now shine,
Faster builds arise.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This PR updates dependencies and restructures the lint-test pipeline. The changes seem reasonable, but I have a few comments regarding dependency management and code clarity.

Summary of Findings

• Dependency Versioning: Consider using more specific version constraints for dependencies to ensure consistent behavior across different environments. While upgrading to the latest versions is good, pinning to a specific minor or patch version can prevent unexpected issues due to breaking changes in future releases.
• Clarity of Dependency Changes: It would be helpful to include a more detailed explanation of why specific dependencies were upgraded or changed, especially for the llm dependencies. This helps maintainers understand the rationale behind the changes and potential impacts.
• FIXME Comment: The removal of the version constraint on anyio is concerning given the FIXME comment. Ensure that the hanging tests issue has been resolved before removing the pin.

Merge Readiness

The pull request updates dependencies and restructures the pipeline. While the changes seem beneficial, the anyio dependency change needs further investigation. I recommend ensuring that the hanging tests issue is resolved before merging. I am unable to approve this pull request, and other reviewers should review and approve this code before merging.

[codecov]

Codecov Report

Attention: Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
plugboard/_zmq/zmq_proxy.py	88.88%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[chrisk314]

@toby-coleman I've bumped the dependency versions in the lockfile. Noticed that Ray has a more recent version whilst working on the cluster. I've also split the lint-test pipeline into multiple jobs - now takes ~190s vs ~330s previously.

[toby-coleman]

Looks good to me. Couple of other things for the future:

• The next version of Ray will have Python 3.13 support, which will let us close #29.
• Noticed that a mypy replacement is in the works, so hopefully we will get a faster replacement for that!

[chrisk314]

Noticed that a mypy replacement is in the works, so hopefully we will get a faster replacement for that!

This is exciting news! Much needed.

[chrisk314]

This PR was missing a change to a test file which was caught after merge. Main was updated directly and force pushed. Actual squash commit is daf3ac4.