Deprecation Notice: ray_lightning to be Replaced with New LightningTrainer in Ray 2.4

[woshiyyya]

Dear ray_lightning users and community members,

We wanted to share some important news regarding the future of the ray_lightning library. As you know, ray_lightning has been a valuable library for enabling distributed training with PyTorch Lightning using the Ray distributed computing framework. It has provided an easy way for parallel model training on Ray while still leveraging the features of PyTorch Lightning.

However, as the PyTorch Lightning and Ray ecosystems have evolved, ray_lightning has struggled to keep pace with the latest versions of both libraries. As a result, it has become less integrated with the Ray ecosystem and its features.

We are excited to announce that, starting from Ray release 2.4, we are introducing a new LightningTrainer, which natively integrates with Ray AI Runtime (Ray AIR). The LightningTrainer will replace ray_lightning and offer several key improvements:

• Support full features of PyTorch Lightning with only a few code changes.
• Support multi-node multi-GPU Distributed Data Parallel (DDP) training on Ray clusters.
• Faster data preprocessing and batch inference with Ray Data.
• Seamless integration with Ray Tune for experiment management and hyper-parameter tuning.

We will no longer maintain ray_lightning, and the last supported versions are pytorch_lightning==1.6.4 and ray==2.4.0. We encourage users to start migrating to the new LightningTrainer to take full advantage of the improved features and integration with the Ray ecosystem. For more information on how to get started with the new LightningTrainer and migrate your existing ray_lightning projects, please refer to our official documentation

LightningTrainer examples:

• [Basic] Train an Image Classifier with LightningTrainer
• [Advanced] Fine-tune a BERT Text classifier with LightningTrainer + Ray Data
• [Advanced] Hyper-parameter Tuning with LightningTrainer and Ray Tune.
• [Expert] Fine-tune dolly-v2-7b with AIR LightningTrainer and FSDP

We appreciate your support and understanding as we continue to improve and innovate within the Ray communities. If you have any questions or concerns, please don't hesitate to reach out to our team through the Ray forums or GitHub issues.

Thank you for being a part of our community!

Best regards,
The Ray Teams

[hanslovsky]

It would be a good idea to add this deprecation notice to the README of this repo. I tried unsuccessfully to run ray_lightning with ray 2.7 until I found this issue. I was able to run a basic example with LightningTrainer, so having the link in the README would be very helpful. I can make a PR if you like.