From f1fe94481e6357323ea4e8f42ce2b8235940a8df Mon Sep 17 00:00:00 2001 From: awaelchli Date: Thu, 1 Feb 2024 16:19:33 +0100 Subject: [PATCH 1/6] fix --- docs/source-fabric/advanced/compile.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source-fabric/advanced/compile.rst b/docs/source-fabric/advanced/compile.rst index 3e47991675b54..000e9086994e0 100644 --- a/docs/source-fabric/advanced/compile.rst +++ b/docs/source-fabric/advanced/compile.rst @@ -137,7 +137,7 @@ You can check whether your model produces graph breaks by calling ``torch.compil # Force an error if there is a graph break in the model model = torch.compile(model, fullgraph=True) -Be aware that the error messages produced here are often quite cryptic, so you will likely have to do some `troubleshooting `_ to fully optimize your model. +Be aware that the error messages produced here are often quite cryptic, so you will likely have to do some `troubleshooting `_ to fully optimize your model. ---- From dceca7907f5e6e98e635db6b658ddb2a6f4567e7 Mon Sep 17 00:00:00 2001 From: awaelchli Date: Sat, 3 Feb 2024 05:29:41 +0100 Subject: [PATCH 2/6] version --- src/lightning/fabric/CHANGELOG.md | 4 ++-- src/lightning/pytorch/CHANGELOG.md | 2 +- src/version.info | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/lightning/fabric/CHANGELOG.md b/src/lightning/fabric/CHANGELOG.md index 89783a7db5ccd..f268070ca84be 100644 --- a/src/lightning/fabric/CHANGELOG.md +++ b/src/lightning/fabric/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). -## [2.2.0] - 2024-02-XX +## [2.2.0] - 2024-02-06 ### Added @@ -15,7 +15,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). - Added `TransformerEnginePrecision(fallback_compute_dtype=)` to control the dtype of operations that don't support fp8 ([#19082](https://github.com/Lightning-AI/lightning/pull/19082)) - Added support for clipping gradients by value with FSDP ([#19236](https://github.com/Lightning-AI/lightning/pull/19236)) - Added a utility function and CLI to consolidate FSDP sharded checkpoints into a single file ([#19213](https://github.com/Lightning-AI/lightning/pull/19213)) -- (Experimental) Added support for re-compiling the model inside `Fabric.setup()` over the FSDP/DDP wrappers ([#19280](https://github.com/Lightning-AI/lightning/pull/19280)) +- Added support for re-compiling the model inside `Fabric.setup()` over the FSDP/DDP wrappers ([#19280](https://github.com/Lightning-AI/lightning/pull/19280)) ### Changed diff --git a/src/lightning/pytorch/CHANGELOG.md b/src/lightning/pytorch/CHANGELOG.md index 57b9c926e8d70..efaaa19764877 100644 --- a/src/lightning/pytorch/CHANGELOG.md +++ b/src/lightning/pytorch/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). -## [2.2.0] - 2024-02-XX +## [2.2.0] - 2024-02-06 ### Added diff --git a/src/version.info b/src/version.info index e2091f21d9da5..ccbccc3dc6263 100644 --- a/src/version.info +++ b/src/version.info @@ -1 +1 @@ -2.2.0.rc0 +2.2.0 From b2e3fa893350e48742fe71924ddccdef8f3b0ba2 Mon Sep 17 00:00:00 2001 From: Jirka Date: Tue, 6 Feb 2024 12:16:37 +0100 Subject: [PATCH 3/6] docs: update chlog after `2.1.4` --- src/lightning/app/CHANGELOG.md | 7 +++++++ src/lightning/data/CHANGELOG.md | 30 +++++++++++++++++++++++++++++- 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/src/lightning/app/CHANGELOG.md b/src/lightning/app/CHANGELOG.md index 99daf11f52322..46e37963fa9e5 100644 --- a/src/lightning/app/CHANGELOG.md +++ b/src/lightning/app/CHANGELOG.md @@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). +## [2.1.4] - 2024-01-31 + +### Changed + +- Remove torch distributed for the Dataset Optimizer ([#19182](https://github.com/Lightning-AI/lightning/pull/19182)) + + ## [2.1.3] - 2023-12-21 ### Changed diff --git a/src/lightning/data/CHANGELOG.md b/src/lightning/data/CHANGELOG.md index 9a61d45eb5d59..2d9410ca4895d 100644 --- a/src/lightning/data/CHANGELOG.md +++ b/src/lightning/data/CHANGELOG.md @@ -5,6 +5,34 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). +## [2.1.4] - 2024-01-31 + +### Added + +- Added support for nested folders in map operator ([#19366](https://github.com/Lightning-AI/lightning/pull/19366)) +- Added support for weights to evenly distributed works among workers for map operator ([#19365](https://github.com/Lightning-AI/lightning/pull/19365)) +- Added profiling support to StreamingDataloader ([#19338](https://github.com/Lightning-AI/lightning/pull/19338)) +- Allow any AWS authentication method in studios ([#19336](https://github.com/Lightning-AI/lightning/pull/19336)) +- Added walk operator #19333 +- Added intra node shuffling to accelerate second epoch in StreamingDataset ([#19296](https://github.com/Lightning-AI/lightning/pull/19296)) +- Enabled map over inputs without files input ([#19285](https://github.com/Lightning-AI/lightning/pull/19285)) +- Added Fault Tolerance v2 ([#19196](https://github.com/Lightning-AI/lightning/pull/19196), [#19201](https://github.com/Lightning-AI/lightning/pull/19201)) + +### Changed + +- Switched map operator arguments order ([#19345](https://github.com/Lightning-AI/lightning/pull/19345)) +- Removed torch distributed for the Dataset Optimizer ([#19182](https://github.com/Lightning-AI/lightning/pull/19182)) +- Remove `__len__` from CombinedStreamingDataset ([#19321](https://github.com/Lightning-AI/lightning/pull/19321)) + +### Fixed + +- Fixed race condition in downloader ([#19348](https://github.com/Lightning-AI/lightning/pull/19348)) +- Fixed serializer `io.bytes` image in JPEGSerializer ([#19369](https://github.com/Lightning-AI/lightning/pull/19369)) +- Fixed several bugs found in Studio Data Processor ([#19309](https://github.com/Lightning-AI/lightning/pull/19309)) +- Fixed handling queue errors in streaming dataset reader ([#19167](https://github.com/Lightning-AI/lightning/pull/19167)) +- Fixed chunks eviction in StreamingDataset ([#19214](https://github.com/Lightning-AI/lightning/pull/19214)) + + ## [2.1.3] - 2023-12-21 ### Added @@ -26,7 +54,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). - Make input dir in `DataProcessor` required ([#18910](https://github.com/Lightning-AI/lightning/pull/18910)) - Remove the `LightningDataset` relying on un-maintained torchdata ([#19019](https://github.com/Lightning-AI/lightning/pull/19019)) -# Fixed +### Fixed - Resolve checkpointing for the Streaming Dataset ([#19123](https://github.com/Lightning-AI/lightning/pull/19123)) - Resolve Item Loader bugs ([#19017](https://github.com/Lightning-AI/lightning/pull/19017)) From 824e98d16745649db43c2e5bfe142f2815f402b1 Mon Sep 17 00:00:00 2001 From: Jirka Date: Tue, 6 Feb 2024 13:55:15 +0100 Subject: [PATCH 4/6] not good but was asked to do :( --- src/lightning/data/CHANGELOG.md | 30 +----------------------------- 1 file changed, 1 insertion(+), 29 deletions(-) diff --git a/src/lightning/data/CHANGELOG.md b/src/lightning/data/CHANGELOG.md index 2d9410ca4895d..9a61d45eb5d59 100644 --- a/src/lightning/data/CHANGELOG.md +++ b/src/lightning/data/CHANGELOG.md @@ -5,34 +5,6 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). -## [2.1.4] - 2024-01-31 - -### Added - -- Added support for nested folders in map operator ([#19366](https://github.com/Lightning-AI/lightning/pull/19366)) -- Added support for weights to evenly distributed works among workers for map operator ([#19365](https://github.com/Lightning-AI/lightning/pull/19365)) -- Added profiling support to StreamingDataloader ([#19338](https://github.com/Lightning-AI/lightning/pull/19338)) -- Allow any AWS authentication method in studios ([#19336](https://github.com/Lightning-AI/lightning/pull/19336)) -- Added walk operator #19333 -- Added intra node shuffling to accelerate second epoch in StreamingDataset ([#19296](https://github.com/Lightning-AI/lightning/pull/19296)) -- Enabled map over inputs without files input ([#19285](https://github.com/Lightning-AI/lightning/pull/19285)) -- Added Fault Tolerance v2 ([#19196](https://github.com/Lightning-AI/lightning/pull/19196), [#19201](https://github.com/Lightning-AI/lightning/pull/19201)) - -### Changed - -- Switched map operator arguments order ([#19345](https://github.com/Lightning-AI/lightning/pull/19345)) -- Removed torch distributed for the Dataset Optimizer ([#19182](https://github.com/Lightning-AI/lightning/pull/19182)) -- Remove `__len__` from CombinedStreamingDataset ([#19321](https://github.com/Lightning-AI/lightning/pull/19321)) - -### Fixed - -- Fixed race condition in downloader ([#19348](https://github.com/Lightning-AI/lightning/pull/19348)) -- Fixed serializer `io.bytes` image in JPEGSerializer ([#19369](https://github.com/Lightning-AI/lightning/pull/19369)) -- Fixed several bugs found in Studio Data Processor ([#19309](https://github.com/Lightning-AI/lightning/pull/19309)) -- Fixed handling queue errors in streaming dataset reader ([#19167](https://github.com/Lightning-AI/lightning/pull/19167)) -- Fixed chunks eviction in StreamingDataset ([#19214](https://github.com/Lightning-AI/lightning/pull/19214)) - - ## [2.1.3] - 2023-12-21 ### Added @@ -54,7 +26,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). - Make input dir in `DataProcessor` required ([#18910](https://github.com/Lightning-AI/lightning/pull/18910)) - Remove the `LightningDataset` relying on un-maintained torchdata ([#19019](https://github.com/Lightning-AI/lightning/pull/19019)) -### Fixed +# Fixed - Resolve checkpointing for the Streaming Dataset ([#19123](https://github.com/Lightning-AI/lightning/pull/19123)) - Resolve Item Loader bugs ([#19017](https://github.com/Lightning-AI/lightning/pull/19017)) From eccb428969b1fdc1177568b42d92e67f1ed9f7e4 Mon Sep 17 00:00:00 2001 From: Jirka Date: Tue, 6 Feb 2024 13:52:30 +0100 Subject: [PATCH 5/6] docs: drop chlog for data --- src/lightning/data/CHANGELOG.md | 79 --------------------------------- 1 file changed, 79 deletions(-) delete mode 100644 src/lightning/data/CHANGELOG.md diff --git a/src/lightning/data/CHANGELOG.md b/src/lightning/data/CHANGELOG.md deleted file mode 100644 index 9a61d45eb5d59..0000000000000 --- a/src/lightning/data/CHANGELOG.md +++ /dev/null @@ -1,79 +0,0 @@ -# Changelog - -All notable changes to this project will be documented in this file. - -The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). - - -## [2.1.3] - 2023-12-21 - -### Added - -- Add fault tolerance `StreamingDataset` ([#19052](https://github.com/Lightning-AI/lightning/pull/19052)) -- Add numpy support for the `StreamingDataset` ([#19050](https://github.com/Lightning-AI/lightning/pull/19050)) -- Add fault tolerance for the `StreamingDataset` ([#19049](https://github.com/Lightning-AI/lightning/pull/19049)) -- Add direct s3 support to the `StreamingDataset` ([#19044](https://github.com/Lightning-AI/lightning/pull/19044)) -- Add disk usage check before downloading files ([#19041](https://github.com/Lightning-AI/lightning/pull/19041)) - -### Changed - -- Cleanup chunks right away if the dataset doesn't fit within the cache in `StreamingDataset` ([#19168](https://github.com/Lightning-AI/lightning/pull/19168)) -- `StreamingDataset` improve deletion strategy ([#19118](https://github.com/Lightning-AI/lightning/pull/19118)) -- Improve `StreamingDataset` Speed ([#19114](https://github.com/Lightning-AI/lightning/pull/19114)) -- Remove time in the Data Processor progress bar ([#19108](https://github.com/Lightning-AI/lightning/pull/19108)) -- Optimize loading time for chunks to be there ([#19109](https://github.com/Lightning-AI/lightning/pull/19109)) -- Resolve path for `StreamingDataset` ([#19094](https://github.com/Lightning-AI/lightning/pull/19094)) -- Make input dir in `DataProcessor` required ([#18910](https://github.com/Lightning-AI/lightning/pull/18910)) -- Remove the `LightningDataset` relying on un-maintained torchdata ([#19019](https://github.com/Lightning-AI/lightning/pull/19019)) - -# Fixed - -- Resolve checkpointing for the Streaming Dataset ([#19123](https://github.com/Lightning-AI/lightning/pull/19123)) -- Resolve Item Loader bugs ([#19017](https://github.com/Lightning-AI/lightning/pull/19017)) - - -## [2.1.2] - 2023-11-15 - -### Added - -- Created cache dir if it doesn't exist ([#18955](https://github.com/Lightning-AI/lightning/pull/18955)) -- Cached directory per worker to avoid collisions ([#18957](https://github.com/Lightning-AI/lightning/pull/18957)) -- Added the input_dir in the cache_dir to avoid overlapping downloads ([#18960](https://github.com/Lightning-AI/lightning/pull/18960)) -- Added support for deleting chunks ([#18959](https://github.com/Lightning-AI/lightning/pull/18959)) -- Added Video/Audio support ([#18977](https://github.com/Lightning-AI/lightning/pull/18977)) -- Added multiple uploaders to the map, optimize ([#18989](https://github.com/Lightning-AI/lightning/pull/18989)) - -### Changed - -- Greedily select files for data processor workers based on size ([#18907](https://github.com/Lightning-AI/lightning/pull/18907)) -- Prevented downloading more chunks than needed ([#18964](https://github.com/Lightning-AI/lightning/pull/18964)) - - -## [2.1.1] - 2023-11-06 - -### Added - -- Added name and version ([#18796](https://github.com/Lightning-AI/lightning/pull/18796)) -- Added support for text ([#18807](https://github.com/Lightning-AI/lightning/pull/18807)) -- Introduced Dataset Optimizer ( - [#18788](https://github.com/Lightning-AI/lightning/pull/18788), - [#18817](https://github.com/Lightning-AI/lightning/pull/18817), - [#18827](https://github.com/Lightning-AI/lightning/pull/18827) -) -- Added distributed support for StreamingDataset ([#18850](https://github.com/Lightning-AI/lightning/pull/18850)) -- Added broadcast to Dataset Optimizer with multiple nodes ([#18860](https://github.com/Lightning-AI/lightning/pull/18860)) -- Improved Streaming Dataset API ([#18882](https://github.com/Lightning-AI/lightning/pull/18882)) -- Prevent leaking the thread to the workers ([#18891](https://github.com/Lightning-AI/lightning/pull/18891)) -- Added DataRecipe ([#18892](https://github.com/Lightning-AI/lightning/pull/18892)) -- Improved map and chunkify ([#18901](https://github.com/Lightning-AI/lightning/pull/18901)) -- Added human-readable format for chunk_bytes ([#18925](https://github.com/Lightning-AI/lightning/pull/18925)) -- Improved s3 client support ([#18920](https://github.com/Lightning-AI/lightning/pull/18920)) -- Added dataset creation ([#18940](https://github.com/Lightning-AI/lightning/pull/18940)) - - -## [2.1.0] - 2023-10-11 - -### Added - -- Added `LightningDataset` for optimized data loading including fast loading for S3 buckets. ([#17743](https://github.com/Lightning-AI/lightning/pull/17743)) -- Added `LightningIterableDataset` for resumable dataloading with iterable datasets ([#17998](https://github.com/Lightning-AI/lightning/pull/17998)) From cf832a67dc3f5d957be5d753ca0f162cc076dd2d Mon Sep 17 00:00:00 2001 From: awaelchli Date: Wed, 7 Feb 2024 22:30:37 +0100 Subject: [PATCH 6/6] update --- src/lightning/fabric/CHANGELOG.md | 2 +- src/lightning/pytorch/CHANGELOG.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lightning/fabric/CHANGELOG.md b/src/lightning/fabric/CHANGELOG.md index f268070ca84be..ab2e568b7db61 100644 --- a/src/lightning/fabric/CHANGELOG.md +++ b/src/lightning/fabric/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). -## [2.2.0] - 2024-02-06 +## [2.2.0] - 2024-02-08 ### Added diff --git a/src/lightning/pytorch/CHANGELOG.md b/src/lightning/pytorch/CHANGELOG.md index ff2e02bda8957..12f8beb648f35 100644 --- a/src/lightning/pytorch/CHANGELOG.md +++ b/src/lightning/pytorch/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/). -## [2.2.0] - 2024-02-06 +## [2.2.0] - 2024-02-08 ### Added