Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MXNet example as a plugin to OpenFL #349

Merged
merged 9 commits into from Mar 1, 2022
Merged

MXNet example as a plugin to OpenFL #349

merged 9 commits into from Mar 1, 2022

Conversation

ViktoriiaRomanova
Copy link
Contributor

@ViktoriiaRomanova ViktoriiaRomanova commented Feb 24, 2022 •
edited

  • Added mxnet tutorial and adapter
  • This is the first version, it works for CPU and GPU.

@ViktoriiaRomanova
example how to add framework support
38a960e 
 added mxnet tutorial and adapter
@ViktoriiaRomanova ViktoriiaRomanova linked an issue Feb 24, 2022 that may be closed by this pull request
Non-TF/PyTorch example as a plugin to OpenFL #176
Closed
@ViktoriiaRomanova ViktoriiaRomanova marked this pull request as draft February 24, 2022 00:32
@ViktoriiaRomanova
Copy link
Contributor Author

Jenkins please retry a build

@alexey-gruzdev alexey-gruzdev added sample enhancement New feature or request labels Feb 24, 2022
@ViktoriiaRomanova ViktoriiaRomanova marked this pull request as ready for review February 25, 2022 07:24
dmitryagapov
dmitryagapov suggested changes Feb 25, 2022
View reviewed changes
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/envoy_config_one.yaml Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/mxnet_adapter.py Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/mxnet_adapter.py Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/mxnet_adapter.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/mxnet_adapter.py Outdated Show resolved Hide resolved
alexey-khorkin
alexey-khorkin reviewed Feb 25, 2022
View reviewed changes
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
alexey-khorkin
alexey-khorkin suggested changes Feb 25, 2022
View reviewed changes
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/sd_requirements.txt Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/sd_requirements.txt Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/requirements.txt Outdated Show resolved Hide resolved
@ViktoriiaRomanova
changed the way set and get optimizer state in mxnet_adapter
4acde32 
edited README
add cuda monitor plugin
edited list libraries in sd_requirements and requirements
edited shard_descriptor
alexey-khorkin
alexey-khorkin suggested changes Feb 28, 2022
View reviewed changes
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/mxnet_adapter.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/mxnet_adapter.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/workspace/mxnet_adapter.py Outdated Show resolved Hide resolved
openfl-tutorials/interactive_api/MXNet_landmarks/README.md Show resolved Hide resolved
@ViktoriiaRomanova
edited README
273381d 
edited list libraries requirements
Minor fixes shard descriptor and mxnet adapter
@dmitryagapov dmitryagapov mentioned this pull request Feb 28, 2022
Open
igor-davidyuk
igor-davidyuk reviewed Mar 1, 2022
View reviewed changes
openfl-tutorials/interactive_api/MXNet_landmarks/envoy/landmark_shard_descriptor.py
if self._target_shape[0] != '1':
raise ValueError('Target has a wrong shape')

def process_data(self, name_csv_file) -> None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do processing in runtime, without saving additional files?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can, but then we have to process csv file every time we start the experiment and keep all images in RAM. I guess processing them just once is better.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand the situation correctly, there is a single csv file with all the labels, and you read and split it in at Envoy start time.
In this situation, you do not need to save separate labels to disk, just keep them in memory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are right, we can do it. I just thought that will be easier to process them the same way as a picture.

openfl-tutorials/interactive_api/MXNet_landmarks/README.md Outdated Show resolved Hide resolved
@ViktoriiaRomanova @igor-davidyuk
Update openfl-tutorials/interactive_api/MXNet_landmarks/README.md
5b9f8a2 
Co-authored-by: Igor Davidyuk <igor.davidyuk@intel.com>
@alexey-gruzdev alexey-gruzdev added this to the v1.3 milestone Mar 1, 2022
@alexey-gruzdev alexey-gruzdev merged commit df22c83 into securefederatedai:develop Mar 1, 2022
@github-actions github-actions bot locked and limited conversation to collaborators Mar 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@dmitryagapov dmitryagapov dmitryagapov approved these changes

@aleksandr-mokrov aleksandr-mokrov aleksandr-mokrov approved these changes

@igor-davidyuk igor-davidyuk igor-davidyuk approved these changes

@alexey-khorkin alexey-khorkin alexey-khorkin approved these changes

@alexey-gruzdev alexey-gruzdev Awaiting requested review from alexey-gruzdev

@itrushkin itrushkin Awaiting requested review from itrushkin

Assignees
No one assigned
Labels
enhancement New feature or request sample
Projects
None yet
Milestone
v1.3
Development

Successfully merging this pull request may close these issues.

Non-TF/PyTorch example as a plugin to OpenFL
6 participants
@ViktoriiaRomanova @dmitryagapov @aleksandr-mokrov @igor-davidyuk @alexey-khorkin @alexey-gruzdev