Merge pull request #1927 from AdeelH/backport

[BACKPORT] Backport changes to the `0.21` branch for `v0.21.2` release

.github/workflows/continuous_integration.yml

@@ -14,7 +14,7 @@ jobs:
       DOCKER_BUILDKIT: 1
       IMAGE_TYPE: ${{ matrix.image_type }}
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
 
       - run: df -hT
 
@@ -36,4 +36,4 @@ jobs:
 
       - run: ./scripts/test "coverage"
 
-      - uses: codecov/codecov-action@v2
+      - uses: codecov/codecov-action@v3

.github/workflows/release.yml

@@ -21,7 +21,7 @@ jobs:
       DOCKER_BUILDKIT: 1
       IMAGE_TYPE: ${{ matrix.image_type }}
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
 
       - run: ./scripts/cibuild
 
@@ -35,7 +35,7 @@ jobs:
 
       - run: ./scripts/test "coverage"
 
-      - uses: codecov/codecov-action@v2
+      - uses: codecov/codecov-action@v3
 
       - run: ./scripts/cipublish
         env:

docs/framework/pipelines.rst

@@ -105,7 +105,48 @@ Below we describe some components of the pipeline that you might directly or ind
 
 A lot of these will be familiar from :doc:`../usage/basics`, but note that when configuring a pipeline, instead of dealing with the classes directly, you will instead be configuring their ``Config`` counterparts.
 
-The following table shows the corresponding ``Configs`` for various commonly used classes.
+Configs
+^^^^^^^
+
+A :class:`.Config` is a Raster Vision class based on the pydantic :class:`.BaseModel` class that allows various kinds of configurations to be stored in a systematic, typed, validatable, and serializable way. Most of these also implement a :meth:`build() <.Config.build>` method that allows the corresponding object to be created based on the configuration. For example, :meth:`.RasterioSourceConfig.build` builds a :class:`.RasterioSource` object.
+
+.. note::
+
+   **Configs and backward compatibility**
+
+   Another crucial role that ``Configs`` play is enabling backward compatibility. Suppose you trained a model and stored it in a :ref:`model-bundle <bundle command>` using an older version of Raster Vision, and now want to use that bundle with a newer version of Raster Vision installed. This can be a problem if the specification of any ``Configs`` has changed between the two versions (e.g. if a field was removed or renamed), which means the newer version will not be able to deserialize the older pipeline config stored in the bundle. 
+
+   Raster Vision solves this issue by associating each Raster Vision plugin with a version number (this is distinct from the Python package version) and providing a config-upgrader mechanism. You can define an upgrader function that takes as input the serialized config dict and a version number and modifies the dict in such a way that makes it compatible with the current version. This function is called multiple times for each config--once for each version number, from zero to the current version. An example upgrader function is shown below.
+
+   .. code-block:: python
+
+      def rs_config_upgrader(cfg_dict: dict, version: int) -> dict:
+         if version == 6:
+            # removed in version 7
+            if cfg_dict.get('extent_crop') is not None:
+                  raise ConfigError('RasterSourceConfig.extent_crop is deprecated.')
+            try:
+                  del cfg_dict['extent_crop']
+            except KeyError:
+                  pass
+         elif version == 9:
+            # renamed in version 10
+            cfg_dict['bbox'] = cfg_dict.get('extent')
+            try:
+                  del cfg_dict['extent']
+            except KeyError:
+                  pass
+         return cfg_dict
+
+   This upgrader function can then be registered against the corresponding Config by passing it to the ``upgrader=`` keyword argument in :func:`.register_config` as shown below.
+
+   .. code-block:: python
+
+      @register_config('raster_source', upgrader=rs_config_upgrader)
+      class RasterSourceConfig(Config):
+         ...
+
+The following table shows the corresponding ``Config`` counterparts for various commonly used classes.
 
 .. currentmodule:: rastervision.core
 

docs/release.rst

@@ -66,69 +66,44 @@ Minor or Major Version Release
 
         pip install twine
 
-    To store settings for PyPI you can setup a ``~/.pypirc`` file containing:
+    To store settings for PyPI you can set up a ``~/.pypirc`` file containing:
 
     .. code-block:: console
 
         [pypi]
         username = azavea
 
-    Once packages are published they cannot be changed so be careful. (It's possible to practice using testpypi.) Navigate to the ``raster-vision`` repo on your local filesystem. With the version branch checked out, run something like the following to publish each plugin, and then the top-level package.
-
-    .. code-block:: console
-
-        export RV="/Users/lfishgold/projects/raster-vision"
-
-    .. code-block:: console
+        [testpypi]
+        username = azavea
 
-        cd $RV/rastervision_pipeline
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
+    Once packages are published they cannot be changed, so be careful. (It's possible to practice using TestPyPI.) Navigate to the repo's root directory on your local filesystem. With the version branch checked out, run the following scripts to build packages and publish to PyPI. 
+
+    Build:
 
     .. code-block:: console
 
-        cd $RV/rastervision_aws_batch
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
+        scripts/pypi_build
 
-    .. code-block:: console
-
-        cd $RV/rastervision_aws_s3
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
+    Publish to TestPyPI. (You will be prompted for the PyPI password multiple times--once for each package.)
 
     .. code-block:: console
 
-        cd $RV/rastervision_core
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
-
-    .. code-block:: console
+        scripts/pypi_publish --test
 
-        cd $RV/rastervision_pytorch_learner
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
+    You can then test it with ``pip`` like so:
 
     .. code-block:: console
 
-        cd $RV/rastervision_pytorch_backend
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
-
-    .. code-block:: console
+        pip install --index-url https://test.pypi.org/simple/ rastervision
 
-        cd $RV/rastervision_gdal_vsi
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
+    Finally, if everything looks okay, publish to Pypi.  (You will be prompted for the PyPI password multiple times--once for each package.)
 
     .. code-block:: console
 
-        cd $RV
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
+        scripts/pypi_publish
 
-#.  Announce new release in our `forum <https://github.com/azavea/raster-vision/discussions>`_, and with blog post if it's a big release.
-#.  Make a PR to the master branch that updates the version number to the next development version. For example, if the last release was ``0.20.1``, update the version to ``0.20.2-dev``.
+#.  Announce the new release in our `forum <https://github.com/azavea/raster-vision/discussions>`_, and with a blog post if it's a big release.
+#.  Make a PR to the master branch that updates the version number to the next development version, ``X.Y.Z-dev``. For example, if the last release was ``0.20.1``, update the version to ``0.20.2-dev``.
 
 Bug Fix Release
 -----------------

rastervision_core/rastervision/core/data/utils/misc.py

@@ -3,6 +3,7 @@
 
 import numpy as np
 from PIL import ImageColor
+from skimage.io import imsave
 
 from rastervision.core.box import Box
 
@@ -240,3 +241,8 @@ def ensure_json_serializable(obj: Any) -> dict:
     if isinstance(obj, Box):
         return obj.to_dict()
     return obj
+
+
+def save_img(im_array: np.ndarray, output_path: str):
+    """Save numpy array as image file."""
+    imsave(output_path, im_array)

rastervision_core/rastervision/core/utils/misc.py

@@ -1,12 +1,3 @@
 from pydantic import confloat
 
-import imageio
-import logging
-
 Proportion = confloat(ge=0, le=1)
-
-log = logging.getLogger(__name__)
-
-
-def save_img(im_array, output_path):
-    imageio.imwrite(output_path, im_array)

rastervision_core/requirements.txt

@@ -1,16 +1,15 @@
 rastervision_pipeline==0.21.1
-
 shapely==2.0.1
 geopandas==0.13.2
-
 numpy==1.25.0
 pillow==9.3.0
 pyproj==3.4.0
 rasterio==1.3.7
-imageio==2.22.1
 pystac==1.6.1
 scikit-learn==1.2.2
 scipy==1.10.1
 opencv-python-headless==4.6.0.66
 tqdm==4.65.0
 xarray==2023.2.0
+scikit-image==0.21.0
+boto3==1.28.8