New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Interactive API Experiment - Pytorch Re-ID on Market #156
Merged
alexey-gruzdev
merged 50 commits into
securefederatedai:develop
from
katerina-merkulova:develop
Aug 31, 2021
Merged
Changes from 17 commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
926c217
add re-id experiment with pytorch
katerina-merkulova 9132dc1
add re-id experiment with pytorch
katerina-merkulova 41c77af
Create global variables for CA files/paths (#150)
dmitryagapov 83589b3
fix lint
katerina-merkulova beaa8a9
fix lint
katerina-merkulova 38bac04
@alexey-gruzdev has signed the CLA from Pull Request #156
github-actions[bot] 471c082
fixes after review
katerina-merkulova 6c4879e
Full traceback call stack on broad exception handling. (#157)
aleksandr-mokrov e3c2d77
fix interface
katerina-merkulova 77fac58
fix shard decriptor logic
katerina-merkulova 8a97dab
minor fixes
katerina-merkulova db031df
fix getitem logic
katerina-merkulova 8ccf985
fix lint
katerina-merkulova 8ddbaf1
rm cell outputs
katerina-merkulova cd4947d
Configure envoy health check period form director (#153)
aleksandr-mokrov 9f52f3a
add shard descriptor's copy
katerina-merkulova 7474ef0
minor changes
katerina-merkulova 594770e
add downloading
katerina-merkulova 82dbdad
add shard_config key to start_envoy
katerina-merkulova 7840401
fix lint
katerina-merkulova 2140515
fix lint
katerina-merkulova 7ca4f9e
add check if dataset folder exists
katerina-merkulova 847d4dc
improve check if dataset folder exists
katerina-merkulova 4f48131
Tests for interactive API (#151)
itrushkin 44f42cc
Unbalanced dataset splits (#125)
itrushkin 0a695d5
Use epochs instead batch_num. Log current epoch number (#95)
maradionov 57c1068
add re-id experiment with pytorch
katerina-merkulova 186ae44
add re-id experiment with pytorch
katerina-merkulova 344b1e7
fix lint
katerina-merkulova 94c648c
fix lint
katerina-merkulova f2b5a03
fixes after review
katerina-merkulova e8c1d07
fix interface
katerina-merkulova 6230ebc
fix shard decriptor logic
katerina-merkulova 615dd06
minor fixes
katerina-merkulova 48155f5
fix getitem logic
katerina-merkulova bfcf440
fix lint
katerina-merkulova 379ba80
rm cell outputs
katerina-merkulova 3bace4e
add shard descriptor's copy
katerina-merkulova 5bc7305
minor changes
katerina-merkulova d04cbe9
add downloading
katerina-merkulova 045ff68
add shard_config key to start_envoy
katerina-merkulova af390d9
fix lint
katerina-merkulova b2224f7
fix lint
katerina-merkulova 8d50b74
add check if dataset folder exists
katerina-merkulova 555ba6f
improve check if dataset folder exists
katerina-merkulova 177cad0
Merge branch 'develop' of https://github.com/katerina-merkulova/openf…
katerina-merkulova 3b48802
fix requirements
katerina-merkulova 1fc763e
remove repeating downloading
katerina-merkulova f080fba
fix downloading logic
katerina-merkulova d75e1a9
fix lint
katerina-merkulova File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
4 changes: 4 additions & 0 deletions
4
openfl-tutorials/interactive_api/Pytorch_Market_Re-ID/director_folder/director_config.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
settings: | ||
listen_ip: localhost | ||
sample_shape: ['64', '128', '3'] | ||
target_shape: ['2'] |
4 changes: 4 additions & 0 deletions
4
openfl-tutorials/interactive_api/Pytorch_Market_Re-ID/director_folder/start_director.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
#!/bin/bash | ||
set -e | ||
|
||
fx director start --disable-tls -c director_config.yaml |
4 changes: 4 additions & 0 deletions
4
...tutorials/interactive_api/Pytorch_Market_Re-ID/director_folder/start_director_with_tls.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
#!/bin/bash | ||
set -e | ||
FQDN=$1 | ||
fx director start -c director_config.yaml -rc cert/root_ca.crt -pk cert/"${FQDN}".key -oc cert/"${FQDN}".crt |
117 changes: 117 additions & 0 deletions
117
...fl-tutorials/interactive_api/Pytorch_Market_Re-ID/envoy_folder/market_shard_descriptor.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
# Copyright (C) 2020-2021 Intel Corporation | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
"""Market shard descriptor.""" | ||
|
||
import re | ||
import zipfile | ||
from pathlib import Path | ||
|
||
import gdown | ||
from PIL import Image | ||
|
||
from openfl.interface.interactive_api.shard_descriptor import ShardDescriptor | ||
|
||
|
||
class MarketShardDescriptor(ShardDescriptor): | ||
""" | ||
Market1501 Shard descriptor class. | ||
|
||
Reference: | ||
Zheng et al. Scalable Person Re-identification: A Benchmark. ICCV 2015. | ||
URL: http://www.liangzheng.org/Project/project_reid.html | ||
|
||
Dataset statistics: | ||
identities: 1501 (+1 for background) | ||
images: 12936 (train) + 3368 (query) + 15913 (gallery) | ||
""" | ||
|
||
def __init__(self, datafolder: str = 'Market-1501-v15.09.15', | ||
rank_worldsize: str = '1,1') -> None: | ||
"""Initialize MarketShardDescriptor.""" | ||
super().__init__() | ||
|
||
# Settings for sharding the dataset | ||
self.rank, self.worldsize = tuple(int(num) for num in rank_worldsize.split(',')) | ||
|
||
self.download() | ||
self.pattern = re.compile(r'([-\d]+)_c(\d)') | ||
self.dataset_dir = Path.cwd() / datafolder | ||
self.train_dir = self.dataset_dir / 'bounding_box_train' | ||
self.query_dir = self.dataset_dir / 'query' | ||
self.gal_dir = self.dataset_dir / 'bounding_box_test' | ||
self._check_before_run() | ||
|
||
self.train_path = list(self.train_dir.glob('*.jpg'))[self.rank - 1::self.worldsize] | ||
self.query_path = list(self.query_dir.glob('*.jpg'))[self.rank - 1::self.worldsize] | ||
self.gal_path = list(self.gal_dir.glob('*.jpg'))[self.rank - 1::self.worldsize] | ||
|
||
self.mode = 'train' | ||
self.imgs_path = self.train_path | ||
|
||
def set_mode(self, mode='train'): | ||
"""Set mode for getitem.""" | ||
self.mode = mode | ||
if self.mode == 'train': | ||
self.imgs_path = self.train_path | ||
elif self.mode == 'query': | ||
self.imgs_path = self.query_path | ||
elif self.mode == 'gallery': | ||
self.imgs_path = self.gal_path | ||
else: | ||
raise Exception(f'Wrong mode: {mode}') | ||
|
||
def __len__(self): | ||
"""Length of shard.""" | ||
return len(self.imgs_path) | ||
|
||
def __getitem__(self, index: int): | ||
"""Return an item by the index.""" | ||
img_path = self.imgs_path[index] | ||
pid, camid = map(int, self.pattern.search(img_path.name).groups()) | ||
|
||
img = Image.open(img_path) | ||
return img, (pid, camid) | ||
|
||
@property | ||
def sample_shape(self): | ||
"""Return the sample shape info.""" | ||
return ['64', '128', '3'] | ||
|
||
@property | ||
def target_shape(self): | ||
"""Return the target shape info.""" | ||
return ['2'] | ||
|
||
@property | ||
def dataset_description(self) -> str: | ||
"""Return the dataset description.""" | ||
return (f'Market dataset, shard number {self.rank} ' | ||
f'out of {self.worldsize}') | ||
|
||
def _check_before_run(self): | ||
"""Check if all files are available before going deeper.""" | ||
if not self.dataset_dir.exists(): | ||
raise RuntimeError(f'{self.dataset_dir} is not available') | ||
if not self.train_dir.exists(): | ||
raise RuntimeError(f'{self.train_dir} is not available') | ||
if not self.query_dir.exists(): | ||
raise RuntimeError(f'{self.query_dir} is not available') | ||
if not self.gal_dir.exists(): | ||
raise RuntimeError(f'{self.gal_dir} is not available') | ||
|
||
@staticmethod | ||
def download(): | ||
"""Download Market1501 dataset.""" | ||
url = 'https://drive.google.com/uc?id=0B8-rUzbwVRk0c054eEozWG9COHM' | ||
output = 'Market.zip' | ||
gdown.download(url, output, quiet=False) | ||
|
||
with zipfile.ZipFile(output, 'r') as zip_ref: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add a check if the folder already exists There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
zip_ref.extractall(Path.cwd()) | ||
|
||
Path(output).unlink() # remove zip | ||
|
||
|
||
if __name__ == '__main__': | ||
MarketShardDescriptor.download() |
2 changes: 2 additions & 0 deletions
2
openfl-tutorials/interactive_api/Pytorch_Market_Re-ID/envoy_folder/requirements.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
gdown==3.13.0 | ||
Pillow==3.8.1 |
4 changes: 4 additions & 0 deletions
4
openfl-tutorials/interactive_api/Pytorch_Market_Re-ID/envoy_folder/shard_config_one.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
template: market_shard_descriptor.MarketShardDescriptor | ||
params: | ||
datafolder: Market-1501-v15.09.15 | ||
rank_worldsize: 1,2 |
4 changes: 4 additions & 0 deletions
4
openfl-tutorials/interactive_api/Pytorch_Market_Re-ID/envoy_folder/shard_config_two.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
template: market_shard_descriptor.MarketShardDescriptor | ||
params: | ||
datafolder: Market-1501-v15.09.15 | ||
rank_worldsize: 2,2 |
4 changes: 4 additions & 0 deletions
4
openfl-tutorials/interactive_api/Pytorch_Market_Re-ID/envoy_folder/start_envoy.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
#!/bin/bash | ||
set -e | ||
|
||
fx envoy start -n env_one --disable-tls -dh localhost -dp 50051 -sc shard_config_one.yaml |
6 changes: 6 additions & 0 deletions
6
openfl-tutorials/interactive_api/Pytorch_Market_Re-ID/envoy_folder/start_envoy_with_tls.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
#!/bin/bash | ||
set -e | ||
ENVOY_NAME=$1 | ||
DIRECTOR_FQDN=$2 | ||
|
||
fx envoy start -n "$ENVOY_NAME" --shard-config-path shard_config.yaml -d "$DIRECTOR_FQDN":50051 -rc cert/root_ca.crt -pk cert/"$ENVOY_NAME".key -oc cert/"$ENVOY_NAME".crt |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a check if the file already exists
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done