Add storage cache in Apache Arrow format using Datumaro #2009

cih9088 · 2023-04-14T05:03:28Z

Summary

Add storage cache feature for cls, det, seg tasks

Added storage cache using Datumaro.
Make Image class a bit friendly to arrow format.
~~Changed label_entities to dictionary type with integer key.~~

The unit tests would be added soon.

How to test

Checklist

I have added unit tests to cover my changes.
I have added integration tests to cover my changes.
I have added e2e tests for validation.
I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).
I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
I have linked related issues.

License

I submit my code changes under the same Apache License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

otx/api/entities/image.py

wonjuleee

I am not sure the storage caching is probably done in Apache Arrow itself rather than OTX framework. If so, do we need to provide the configurations for storage caching in OTX?

otx/cli/manager/config_manager.py

otx/core/data/adapter/action_dataset_adapter.py

otx/core/data/caching/storage_cache.py

…envinotoolkit#2012) Revert segmentation ignore=True

sungmanc

@cih9088 Could you change the Datumaro version to enable the arrow format? and add unit-tests for this feature?

Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com>

sungmanc · 2023-04-19T05:06:33Z

@wonjuleee, @cih9088 Since all tests are passed, we're going to merge this PR after preparing the validation report (it will be merged by the end of today).

I asked Inhyuk to check w/ Geti since this PR includes small changes to ImageEntity.
We could easily assume there will be no problem, however, sometimes Geti occurs an error even though there are only small changes.

Thanks for the hard work

goodsong81

Generally looks OK to me. I have a few questions including the one below.
Did you check the Geti pre-commit / unit / component test? If not, you can create a dummy PR for Geti to run the continuous-integration/jenkins/pr-head. (It's different from Geti E2E test build).
Please understand that it's due to the change in base entity which might affect Geti.

goodsong81 · 2023-04-19T05:53:27Z

tests/unit/core/data/manager/test_dataset_manager.py

@@ -74,7 +75,13 @@ def test_get_image_path(self, task, subset):
        random_data = DatasetManager.get_image_path(
            generate_datumaro_dataset_item(item_id="0", subset=subset, task=task)
        )
-        assert random_data is not None
+        assert random_data is None


Meaning that this function should return None if not given the temp_dir?

datumaro.Image was refactored to have a separate class implementation depending on sources (file path, numpy, raw binary) in Datumaro v1.2.0. If temp_dir is not given, (data source is not a file), path attribute does not exist in datumaro.Image.

This change makes sure that if path attribute is not found, return None.

cih9088 · 2023-04-19T06:24:01Z

I asked Inhyuk to check w/ Geti since this PR includes small changes to ImageEntity. We could easily assume there will be no problem, however, sometimes Geti occurs an error even though there are only small changes.

Did you check the Geti pre-commit / unit / component test?

I'm waiting for write permission to the repo.

goodsong81 · 2023-04-19T06:29:46Z

I asked Inhyuk to check w/ Geti since this PR includes small changes to ImageEntity. We could easily assume there will be no problem, however, sometimes Geti occurs an error even though there are only small changes.

Did you check the Geti pre-commit / unit / component test?

I'm waiting for write permission to the re

I asked Inhyuk to check w/ Geti since this PR includes small changes to ImageEntity. We could easily assume there will be no problem, however, sometimes Geti occurs an error even though there are only small changes.

Did you check the Geti pre-commit / unit / component test?

I'm waiting for write permission to the repo.

It's for you :)
https://github.com/intel-innersource/frameworks.ai.interactive-ai-workflow.sonoma-creek/pull/7442

-> Passed.

https://ci.iotg.sclab.intel.com/job/IMPT/job/utils-common/job/sc-github-check/job/PR-7442/1/

cih9088 · 2023-04-19T06:56:16Z

-> Passed. ci.iotg.sclab.intel.com/job/IMPT/job/utils-common/job/sc-github-check/job/PR-7442/1

Thanks a lot for your goodwill. 🥇

* HOT-FIX: Revert segmentation model's ignore mode in CLI (#2011) Revert segmentation ignore=True * Improve tiling preprocess (#2013) * prevent timeout during init phase * Fix reg tests (#2008) * Edit regression tests * Change the dataset root * Miss typo * Fix pre-commit * Fix openvino import error due to Tiler init import (#2015) Remove init import for Tiler to prevent OpenVINO import * Bump up version to 1.2.0 (#2017) * Set the python version to "3.10" for code-scan workflow * Add missing __init__.py (#2019) * Add missing __init__.py * Change license * Release 1.2.0rc1 * Fix issue that str2bool not being applied in certain cases (#2023) * Add workaround solution * Fix minor * Remove str int * Fix default dict (#2025) fix: change default to configdict Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com> * Convert dummy datasets to toy datasets (#1988) * Update cls, det datsets * Remove useless files * Change action datasets * Edit action dataset * change dir * Add xml files * Remove useless * Edite tets * Fix tests * Fix tests * Remove ptc * Remove * Fix precommit * Update dataset, fix cls bug * Remove useless dataset * Edit drop_last * Fix missed part * Change threshold values to unifying * bugfix: squeezing to 1 dimenetion * Change threshold for deployment * Fix multi-gpu issue, e2e tests * Decrease num_workers for tiling test and tiling processes * Revert num_workers for tests * Fix datsets --------- Co-authored-by: eunwoosh <eunwoo.shin@intel.com> * Fix E2E tests (#2032) * Optimize data preprocessing time and enhance overall performance in semantic segmentation (#2020) * HOTFIX: change doc version to 1.2.0 * Add storage cache in Apache Arrow format using Datumaro (#2009) * feat: change label entity to dictionay * feat: add datumaro arrow cache * refacor: move to proper directory * fix: align to the latest * fix: align data to otx * fix: align new version * refactor: disable storage cache for action tasks * test: fix * fix: version back * docs: add to changelog * fix: keep __height, __width * docs: add description * test: revert tests * fix: revert back to list * style: ruff * HOT-FIX: Revert segmentation model's ignore mode in CLI (Develop) (#2012) Revert segmentation ignore=True * fix: make force verbose * test: add storage cache test * feat: datumaro 1.2.0 * test: test path exists * test: do deepcopy * style: make black happy --------- Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com> Co-authored-by: Harim Kang <harim.kang@intel.com> * Fix typo in prediction_to_annotation_converter.py (#2028) * HOT-FIX: Revert segmentation model's ignore mode in CLI (Develop) (#2012) Revert segmentation ignore=True * Bump up version to 1.3.0rc0 (#2016) * bug fix * del * revert * revert changlog --------- Co-authored-by: Harim Kang <harim.kang@intel.com> Co-authored-by: Songki Choi <songki.choi@intel.com> * Fix tiling config loading bug (#2030) * fix tiling loading bug * fix mypy * Make multi gpu child processes done right after evaluation (#2033) * Fixes in exportable code (#2031) * Create Actions domain and task type despite flag * Move import torch to the function * Fix str in dump_features * Move dump_frames to otx/api/utils * Remove __pycache__ from dunped exportable code * Add tests for demo --output option * Update sha for exportable code requirements * Add flag to task_type_to_label_domain * Roll back tests creation to add it in separate PR * Add FEATURE_FLAGS_OTX_ACTION_TASKS init in demo.py * Remove extra comments * Fix linter * Add documentation for the noisy label detection feature (#2034) * Add documentation for noisy label detection feature Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Update CHANGELOG.md Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Add documentation Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Fix typo Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Fix small typo --------- Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> Co-authored-by: Songki Choi <songki.choi@intel.com> * Remove skip tests, fix regression tests (#2036) * Remove skip tests, fix regression tests * Fix precommit * Hide internal options from external GUI (#2037) Signed-off-by: Songki Choi <songki.choi@intel.com> * Add unit test for classification task and configurer (#2035) * Reduce the depth of aumix process (#2038) Enable light augmix * Change samples_per_gpu in _infer_model(Detection) (#2041) Change samples_per_gpu in _infer_model * Decrease batch size if CUDA OOM occurs (#2022) * implement adpating bs * refine impl * implement adaptive bs also in cls, seg task * refine adapt bs algo to consider gpu util * refactor code * write comment and docstring * implement decreasig bs on action task * update learning rate after decreasing batch size * implement test code of mmcv automatic_bs file * remove meta modification * remove unused improt * implement test code of torch automatic_bs file * align with pre commit * add line to tell not supporting anomaly * update CHANGELOG * update docs * change argument help * change file name * apply pr comment * add auto_decrease_bs in learning parameters * align with pre commit * fix typo * add integration test * bugfix * update test code * not execute algo in nncf * suppor nncf * apply comment * align with pre commit * change method to set value * refine warning comment * remove breakpoint * make hpo not use auto decrease batch size * refine warning & typo fix * align with pre commit * Update CHANGELOG.md Co-authored-by: Sungman Cho <sungman.cho@intel.com> * update unit test * update integration test * bufix * Release 1.2.0rc2 Signed-off-by: Songki Choi <songki.choi@intel.com> * Update OTX commit hash for exportable code requiements --------- Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com> Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> Signed-off-by: Songki Choi <songki.choi@intel.com> Co-authored-by: Harim Kang <harim.kang@intel.com> Co-authored-by: Eugene Liu <eugene.liu@intel.com> Co-authored-by: Sungman Cho <sungman.cho@intel.com> Co-authored-by: Yunchu Lee <yunchu.lee@intel.com> Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com> Co-authored-by: Inhyuk Cho <andy.inhyuk.jo@intel.com> Co-authored-by: eunwoosh <eunwoo.shin@intel.com> Co-authored-by: Soobee Lee <soobee.lee@intel.com> Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com> Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>

cih9088 added 10 commits April 13, 2023 16:15

feat: change label entity to dictionay

c2b2697

feat: add datumaro arrow cache

002744d

refacor: move to proper directory

cfb440f

fix: align to the latest

1cf7cdd

fix: align data to otx

1deee1f

fix: align new version

72716d7

refactor: disable storage cache for action tasks

6cb1b79

test: fix

c35bf20

fix: version back

f6bec83

docs: add to changelog

b83eb9c

cih9088 added the FEATURE New feature & functionality label Apr 14, 2023

cih9088 added this to the 1.2.0 milestone Apr 14, 2023

cih9088 requested a review from a team as a code owner April 14, 2023 05:03

github-actions bot added ALGO Any changes in OTX Algo Tasks implementation API Any changes in OTX API CLI Any changes in OTE CLI TEST Any changes in tests DOC Improvements or additions to documentation labels Apr 14, 2023

cih9088 force-pushed the feature/arrow branch from 53b524c to b83eb9c Compare April 14, 2023 05:06

fix: keep __height, __width

f993b66

sungmanc reviewed Apr 14, 2023

View reviewed changes

otx/api/entities/image.py Outdated Show resolved Hide resolved

docs: add description

0940054

wonjuleee reviewed Apr 14, 2023

View reviewed changes

otx/api/entities/image.py Show resolved Hide resolved

wonjuleee reviewed Apr 14, 2023

View reviewed changes

cih9088 added 2 commits April 14, 2023 14:40

test: revert tests

53df78d

fix: revert back to list

7b040d1

wonjuleee reviewed Apr 14, 2023

View reviewed changes

otx/cli/manager/config_manager.py Show resolved Hide resolved

otx/core/data/adapter/action_dataset_adapter.py Outdated Show resolved Hide resolved

Merge branch 'develop' into feature/arrow

1d9b45d

cih9088 force-pushed the feature/arrow branch from 9603c9b to f993b66 Compare April 14, 2023 06:11

wonjuleee reviewed Apr 14, 2023

View reviewed changes

otx/core/data/caching/storage_cache.py Outdated Show resolved Hide resolved

otx/core/data/caching/storage_cache.py Show resolved Hide resolved

style: ruff

6faf410

github-actions bot removed the TEST Any changes in tests label Apr 14, 2023

harimkang and others added 2 commits April 14, 2023 09:05

HOT-FIX: Revert segmentation model's ignore mode in CLI (Develop) (op…

c4c0e05

…envinotoolkit#2012) Revert segmentation ignore=True

Merge branch 'develop' into feature/arrow

8776f2d

cih9088 changed the base branch from develop to releases/1.2.0 April 14, 2023 09:09

cih9088 added the RELEASE label Apr 14, 2023

wonjuleee previously approved these changes Apr 14, 2023

View reviewed changes

sungmanc requested changes Apr 17, 2023

View reviewed changes

cih9088 added 5 commits April 19, 2023 08:44

fix: make force verbose

02dff53

Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com>

test: add storage cache test

bbbeb4e

Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com>

feat: datumaro 1.2.0

862f086

Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com>

test: test path exists

09b3495

Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com>

Merge branch 'releases/1.2.0' into feature/arrow

759bfbc

cih9088 dismissed wonjuleee’s stale review via 862f086 April 19, 2023 00:54

github-actions bot added DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM TEST Any changes in tests labels Apr 19, 2023

cih9088 requested a review from sungmanc April 19, 2023 00:55

test: do deepcopy

be52cfb

Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com>

cih9088 force-pushed the feature/arrow branch from 250bf4e to 09b3495 Compare April 19, 2023 01:13

wonjuleee previously approved these changes Apr 19, 2023

View reviewed changes

wonjuleee requested a review from goodsong81 April 19, 2023 01:39

cih9088 dismissed wonjuleee’s stale review via be52cfb April 19, 2023 02:01

style: make black happy

2cbbe2b

Signed-off-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com>

goodsong81 approved these changes Apr 19, 2023

View reviewed changes

goodsong81 merged commit 47d09e2 into openvinotoolkit:releases/1.2.0 Apr 19, 2023
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add storage cache in Apache Arrow format using Datumaro #2009

Add storage cache in Apache Arrow format using Datumaro #2009

cih9088 commented Apr 14, 2023 •

edited

wonjuleee left a comment

sungmanc left a comment

sungmanc commented Apr 19, 2023

goodsong81 left a comment

goodsong81 Apr 19, 2023

cih9088 Apr 19, 2023

cih9088 commented Apr 19, 2023

goodsong81 commented Apr 19, 2023 •

edited

cih9088 commented Apr 19, 2023

Add storage cache in Apache Arrow format using Datumaro #2009

Add storage cache in Apache Arrow format using Datumaro #2009

Conversation

cih9088 commented Apr 14, 2023 • edited

Summary

How to test

Checklist

License

wonjuleee left a comment

Choose a reason for hiding this comment

sungmanc left a comment

Choose a reason for hiding this comment

sungmanc commented Apr 19, 2023

goodsong81 left a comment

Choose a reason for hiding this comment

goodsong81 Apr 19, 2023

Choose a reason for hiding this comment

cih9088 Apr 19, 2023

Choose a reason for hiding this comment

cih9088 commented Apr 19, 2023

goodsong81 commented Apr 19, 2023 • edited

cih9088 commented Apr 19, 2023

cih9088 commented Apr 14, 2023 •

edited

goodsong81 commented Apr 19, 2023 •

edited