Merge back Releases/1.4.0 (#1111)

.github/workflows/pr_check.yml

@@ -49,7 +49,7 @@ jobs:
           python -m pip install tox
       - name: Unit testing
         run: |
-          tox -e tests-py${{ matrix.tox-env-py }}-${{ matrix.tox-env-os }} -- tests/unit --cov
+          tox -e tests-py${{ matrix.tox-env-py }}-${{ matrix.tox-env-os }} --develop -- tests/unit --cov --cov-report=xml
       - name: Integration testing
         run: |
           tox -e tests-py${{ matrix.tox-env-py }}-${{ matrix.tox-env-os }} -- tests/integration

CHANGELOG.md

@@ -9,21 +9,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add tabular data import/export
   (<https://github.com/openvinotoolkit/datumaro/pull/1089>)
 
-## 11/07/2023 - Release 1.4.0rc2
-### New features
-- Add documentation and notebook example for Prune API
-  (<https://github.com/openvinotoolkit/datumaro/pull/1070>)
-
-### Enhancements
-- Give notice that the deprecation works will be done in datumaro==1.5.0
-  (<https://github.com/openvinotoolkit/datumaro/pull/1085>)
-
+## 27/07/2023 - Release 1.4.1
 ### Bug fixes
-- Create cache dir under only writable filesystem
-  (<https://github.com/openvinotoolkit/datumaro/pull/1088>)
+- Report errors for COCO (stream) and Datumaro importers
+  (<https://github.com/openvinotoolkit/datumaro/pull/1110>)
 
-## 07/07/2023 - Release 1.4.0rc1
+## 21/07/2023 - Release 1.4.0
 ### New features
+- Add documentation and notebook example for Prune API
+  (<https://github.com/openvinotoolkit/datumaro/pull/1070>)
 - Changed supported Python version range (>=3.8, <=3.11)
   (<https://github.com/openvinotoolkit/datumaro/pull/1083>)
 - Migrate OpenVINO v2023.0.0
@@ -41,11 +35,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Migrate DVC v3.0.0
   (<https://github.com/openvinotoolkit/datumaro/pull/1072>)
 - Stream dataset import/export
-  (<https://github.com/openvinotoolkit/datumaro/pull/1077>, <https://github.com/openvinotoolkit/datumaro/pull/1081>, <https://github.com/openvinotoolkit/datumaro/pull/1082>, <https://github.com/openvinotoolkit/datumaro/pull/1091>)
+  (<https://github.com/openvinotoolkit/datumaro/pull/1077>, <https://github.com/openvinotoolkit/datumaro/pull/1081>, <https://github.com/openvinotoolkit/datumaro/pull/1082>, <https://github.com/openvinotoolkit/datumaro/pull/1091>, <https://github.com/openvinotoolkit/datumaro/pull/1093>, <https://github.com/openvinotoolkit/datumaro/pull/1098>, <https://github.com/openvinotoolkit/datumaro/pull/1102>)
 - Support mask annotations for CVAT data format
   (<https://github.com/openvinotoolkit/datumaro/pull/1078>)
 
 ### Enhancements
+- Support list query for explorer
+  (<https://github.com/openvinotoolkit/datumaro/pull/1087>)
+- update contributing.md
+  (<https://github.com/openvinotoolkit/datumaro/pull/1094>)
+- Update 3rd-party.txt for release 1.4.0
+  (<https://github.com/openvinotoolkit/datumaro/pull/1099>)
+- Give notice that the deprecation works will be done in datumaro==1.5.0
+  (<https://github.com/openvinotoolkit/datumaro/pull/1085>)
+- Unify COCO, Datumaro, VOC, YOLO importer/exporter progress reporter descriptions
+  (<https://github.com/openvinotoolkit/datumaro/pull/1100>)
 - Enhance import performance for built-in plugins
   (<https://github.com/openvinotoolkit/datumaro/pull/1031>)
 - Change default dtype of load_image() to np.uint8
@@ -64,6 +68,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   (<https://github.com/openvinotoolkit/datumaro/pull/1073>)
 
 ### Bug fixes
+- Create cache dir under only writable filesystem
+  (<https://github.com/openvinotoolkit/datumaro/pull/1088>)
+- Fix: Dataset infos() can be broken if a transform not redefining infos() is stacked on the top
+  (<https://github.com/openvinotoolkit/datumaro/pull/1101>)
 - Fix warnings in test_visualizer.py
   (<https://github.com/openvinotoolkit/datumaro/pull/1039>)
 - Fix LabelMe data format

docs/source/docs/command-reference/context_free/explorer.md

@@ -12,14 +12,18 @@ The command can be applied to a dataset. And if you want to use multiple dataset
 
 Usage:
 ```console
-datum explore [target] -q <path/to/image> or <text_query>
-              [-topk TOPK] [-p PROJECT_DIR] [-s SAVE] [--stage STAGE]
+datum explore [target] [--query-img-path <path/to/image>]
+              [--query-item-id </id/of/image/datasetitem> --query-item-subset <subset/of/image>]
+              [--query-str <text_query>] [-topk TOPK] [-p PROJECT_DIR] [-s SAVE] [--stage STAGE]
 ```
 
 Parameters:
 - `<target>` (string) - Target [dataset revpath](../../user-manual/how_to_use_datumaro.md#dataset-path-concepts).
   By default, prints info about the joined `project` dataset.
-- `-q, --query` (string) - Image path or text to use as query.
+- `--query-img-path` (string) - Image path to use as query or list of it.
+- `--query-item-id` (string) - Datasetitem id of Image to use as query or list of it.
+- `--query-item-subset` (string) - Datasetitem subset of Image to use as query or list of it. (default: None)
+- `--query-str` (string) - Test to use as query or list of it.
 - `-topk` (int) - Number how much you want to find similar data.
 - `-p, --project` (string) - Directory of the project to operate on (default: current directory).
 - `-s, --save` (bool) - Save explorer result files on explore_result folder.
@@ -30,29 +34,39 @@ Parameters:
   and the `project` target, but not intermediate stages). Enabled by default.
 
 Examples:
-- Explore top10 similar images of image query
+- Explore top10 similar images of image path query.
   ```console
-  datum search -q <path/to/image> -topk 10 <path/to/dataset/>
+  datum explore -query-img-path <path/to/image> -topk 10 <path/to/dataset/>
   ```
 
-- Explore top10 similar images of image query within project
+- Explore top10 similar images using image datasetitem id query. Setting the subset of a DatasetItem is not mandatory.
+  ```console
+  datum explore --query-item-id <id/of/image/datasetitem> -topk 10 <path/to/dataset/>
+  ```
+
+- Explore top10 similar images using image datasetitem id and subset query.
+  ```console
+  datum explore --query-item-id <id/of/image/datasetitem> --query-item-subset <subset/of/image/datasetitem> -topk 10 <path/to/dataset/>
+  ```
+
+- Explore top10 similar images of image query within project.
   ```console
   datum project create <...>
   datum project import -f <format> <path/to/dataset/>
-  datum explore -q <path/to/image> -topk 10
+  datum explore -query-img-path <path/to/image> -topk 10
   ```
 
 - Explore top10 similar images of text query, elephant
   ```console
-  datum search -q elephant -topk 10 <path/to/dataset/>
+  datum explore --query-str elephant -topk 10 <path/to/dataset/>
   ```
 
 - Explore top50 similar images of image query list
   ```console
-  datum search -q <path/to/image1> <path/to/image2> <path/to/image3> -topk 50 <path/to/dataset/>
+  datum explore --query-img-path <path/to/image1> <path/to/image2> <path/to/image3> -topk 50 <path/to/dataset/>
   ```
 
 - Explore top50 similar images of text query list
   ```console
-  datum search -q motorcycle bus train -topk 50 <path/to/dataset/>
+  datum explore --query-str motorcycle bus train -topk 50 <path/to/dataset/>
   ```

docs/source/docs/level-up/intermediate_skills/10_data_exploration.rst

@@ -39,17 +39,31 @@ The Python example for the usage of explorer is described in :doc:`here <../../j
 
     .. tab-item:: CLI
 
-        Without the project declaration, we can simply ``explore`` dataset by
+        Without the project declaration, we can simply ``explore`` dataset like below.
+
+        You can set the query using one of the following options: ``QUERY_PATH``, ``QUERY_ID``, or ``QUERY_STR``
 
         .. code-block:: bash
 
-            datum explore <target> --query QUERY -topk TOPK_NUM
+            datum explore <target> --query-img-path QUERY_PATH -topk TOPK_NUM
 
-        ``QUERY`` could be image file path, text description, list of both of them
+        ``QUERY_PATH`` could be image file path or list of them
 
         ``TOPK_NUM`` is an integer that you want to find the number of similar results for query
 
-        Exploration result would be printed by log and result files would be copied into ``explore_result`` folder
+        Exploration result would be printed by log and result files would be copied into ``explore_result`` folder.
+
+        .. code-block:: bash
+
+            datum explore <target> --query-item-id QUERY_ID -topk TOPK_NUM
+
+        ``QUERY_ID`` could be datasetitem id or list of them
+
+        .. code-block:: bash
+
+            datum explore <target> --query-str QUERY_STR -topk TOPK_NUM
+
+        ``QUERY_STR`` could be text description or list of them
 
     .. tab-item:: ProjectCLI
 
@@ -65,14 +79,28 @@ The Python example for the usage of explorer is described in :doc:`here <../../j
 
             datum project import --project <path/to/project> <path/to/data>
 
-        We can ``explore`` similar items for the query
+        We can ``explore`` similar items for the query.
+
+        You can set the query using one of the following options: ``QUERY_PATH``, ``QUERY_ID``, or ``QUERY_STR``
 
         .. code-block:: bash
 
-            datum explore --query QUERY -topk TOPK_NUM -p <path/to/project>
+            datum explore --query-img-path QUERY_PATH -topk TOPK_NUM -p <path/to/project>
 
-        ``QUERY`` could be image file path, text description, list of both of them
+        ``QUERY_PATH`` could be image file path or list of them
 
         ``TOPK_NUM`` is an integer that you want to find the number of similar results for query
 
-        Exploration result would be printed by log and result files would be copied into ``explore_result`` folder
+        Exploration result would be printed by log and result files would be copied into ``explore_result`` folder.
+
+        .. code-block:: bash
+
+            datum explore <target> --query-item-id QUERY_ID -topk TOPK_NUM -p <path/to/project>
+
+        ``QUERY_ID`` could be datasetitem id or list of them
+
+        .. code-block:: bash
+
+            datum explore <target> --query-str QUERY_STR -topk TOPK_NUM -p <path/to/project>
+
+        ``QUERY_STR`` could be text description or list of them

docs/source/docs/release_notes.rst

@@ -4,6 +4,52 @@ Release Notes
 .. toctree::
    :maxdepth: 1
 
+v1.4.1 (2023.07)
+----------------
+- Report errors for COCO (stream) and Datumaro importers
+
+v1.4.0 (2023.07)
+----------------
+
+New features
+^^^^^^^^^^^^
+- Add documentation and notebook example for Prune API
+- Changed supported Python version range (>=3.8, <=3.11)
+- Migrate OpenVINO v2023.0.0
+- Add Roboflow data format support (COCO JSON, Pascal VOC XML, YOLOv5-PyTorch, YOLOv7-PyTorch, YOLOv8, YOLOv5 Oriented Bounding Boxes, Multiclass CSV, TFRecord, CreateML JSON)
+- Add MissingAnnotationDetection transform
+- Add OVMSLauncher
+- Add Prune API
+- Add TritonLauncher
+- Migrate DVC v3.0.0
+- Stream dataset import/export
+- Support mask annotations for CVAT data format
+
+Enhancements
+^^^^^^^^^^^^
+- Support list query for explorer
+- update contributing.md
+- Update 3rd-party.txt for release 1.4.0
+- Give notice that the deprecation works will be done in datumaro==1.5.0
+- Unify COCO, Datumaro, VOC, YOLO importer/exporter progress reporter descriptions
+- Enhance import performance for built-in plugins
+- Change default dtype of load_image() to np.uint8
+- Add OTX ATSS detector model interpreter & refactor interfaces
+- Refactor Launcher and ModelInterpreter
+- Add CVAT data format document
+- Reduce peak memory usage when importing COCO and Datumaro formats
+- Enhance the error message for datum stats to be more user friendly
+- Refactor dataset.py to seperate DatasetStorage
+
+Bug fixes
+^^^^^^^^^
+- Create cache dir under only writable filesystem
+- Fix: Dataset infos() can be broken if a transform not redefining infos() is stacked on the top
+- Fix warnings in test_visualizer.py
+- Fix LabelMe data format
+- Prevent installing protobuf>=4
+- Fix UnionMerge
+
 v1.3.2 (2023.06)
 ----------------
 

setup.py

@@ -19,7 +19,7 @@ def find_version(project_dir=None):
 
     file_path = osp.join(project_dir, "datumaro", "version.py")
 
-    with open(file_path, "r") as version_file:
+    with open(file_path, "r", encoding="utf-8") as version_file:
         version_text = version_file.read()
 
     # PEP440:
@@ -39,7 +39,7 @@ def find_version(project_dir=None):
 
 
 def parse_requirements(filename=CORE_REQUIREMENTS_FILE):
-    with open(filename) as fh:
+    with open(filename, "r", encoding="utf-8") as fh:
         return fh.readlines()
 
 
@@ -51,7 +51,7 @@ def parse_requirements(filename=CORE_REQUIREMENTS_FILE):
 
 DEFAULT_REQUIREMENTS = parse_requirements(DEFAULT_REQUIREMENTS_FILE)
 
-with open("README.md", "r") as fh:
+with open("README.md", "r", encoding="utf-8") as fh:
     long_description = fh.read()
 
 ext_modules = [