Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support encrypted dataset training #2209

Merged
merged 5 commits into from
Jun 1, 2023

Conversation

vinnamkim
Copy link
Contributor

@vinnamkim vinnamkim commented May 31, 2023

Summary

  • Ticket no. 112208
  • Using OTX, users can seamlessly train encrypted datasets created by Datumaro.
  • otx train with --encryption-key CLI argument or ENCRYPTION_KEY env vars enables model training with an encrypted dataset.

How to test

I added tests related to this change.

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@github-actions github-actions bot added CLI Any changes in OTE CLI TEST Any changes in tests DOC Improvements or additions to documentation labels May 31, 2023
@vinnamkim vinnamkim force-pushed the feature/support-encrypted-dataset branch from 188159e to 4be1f3c Compare May 31, 2023 04:41
@vinnamkim vinnamkim marked this pull request as ready for review May 31, 2023 04:42
@vinnamkim vinnamkim requested a review from a team as a code owner May 31, 2023 04:42
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@vinnamkim vinnamkim requested a review from wonjuleee May 31, 2023 04:42
ActionBaseDatasetAdapter signatures

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@vinnamkim vinnamkim added this to the 1.4.0 milestone May 31, 2023
@codecov-commenter
Copy link

codecov-commenter commented May 31, 2023

Codecov Report

Patch coverage: 92.50% and no project coverage change.

Comparison is base (7074d46) 80.24% compared to head (342fe0f) 80.25%.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #2209   +/-   ##
========================================
  Coverage    80.24%   80.25%           
========================================
  Files          469      469           
  Lines        35182    35183    +1     
========================================
+ Hits         28232    28236    +4     
+ Misses        6950     6947    -3     
Flag Coverage Δ
py310 80.25% <92.50%> (+0.01%) ⬆️
py38 80.23% <92.50%> (+<0.01%) ⬆️
py39 80.24% <92.50%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
otx/core/data/adapter/base_dataset_adapter.py 89.01% <90.47%> (+1.57%) ⬆️
otx/cli/manager/config_manager.py 76.96% <92.30%> (+0.55%) ⬆️
otx/cli/tools/train.py 81.89% <100.00%> (+0.15%) ⬆️
otx/core/data/adapter/action_dataset_adapter.py 95.83% <100.00%> (ø)
otx/core/data/adapter/anomaly_dataset_adapter.py 87.12% <100.00%> (ø)
.../core/data/adapter/segmentation_dataset_adapter.py 94.82% <100.00%> (-0.14%) ⬇️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

jaegukhyun
jaegukhyun previously approved these changes May 31, 2023
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Copy link
Contributor

@sungmanc sungmanc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no need to change the Datumaro version?

@vinnamkim
Copy link
Contributor Author

vinnamkim commented May 31, 2023

Is there no need to change the Datumaro version?

DatumaroBinary format and dataset encryption features were added months ago (datumaro==1.1.0).

@goodsong81 goodsong81 merged commit be5ee9b into develop Jun 1, 2023
12 checks passed
@goodsong81 goodsong81 deleted the feature/support-encrypted-dataset branch June 1, 2023 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLI Any changes in OTE CLI DOC Improvements or additions to documentation TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants