Update Part 13b - Secure Classification with Syft Keras and TFE - Sec… #3466
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ure Model Serving.ipynb
|
Check out this pull request on Review Jupyter notebook visual diffs & provide feedback on notebooks. Powered by ReviewNB |
srishilesh
approved these changes
May 8, 2020
There was a problem hiding this comment.
Approving the error fixes in Bengali translations for Part 13 b
Issue resolved #3466
gmuraru
added a commit
to gmuraru/PySyft
that referenced
this pull request
May 13, 2020
* create manifest file to induce pip-dep directory (OpenMined#3420) * Fix typo + warning (OpenMined#3421) * Increase sigmoid exp tolerance (OpenMined#3427) * Fix double tracing bug (OpenMined#3428) * fix bug where actions are duplicated when tracing torch functions on Placeholders * remove unused file * Add an initial of Function Secret Sharing using Plans (OpenMined#3057) * Fix hook by removing unused and confusing functions * Fix hook of PlaceHolder and add tracer to th.tensor * Update list of ambiguous and hook-excluded functions * Improve instantiation of placeholder in Operation response * Fix extract_key bug, add checks and explicative failures * Add tests for plans with comparison and funcs with no args * Add experimental notebooks for back up * Add first version of FSS * Add dtype serialization and small fix in ptr plan * Add checks to AdditiveSharedTensor * Add clarification error to Plan * Add support of multi pointer tensors on plan ptr with>1 locations * Import placeholder instantiation * Update FSS * Run black * Improve FSS * Add a tag index for worker and a find_by_tag method * Clean object storge rm_obj and use tag index * Fix search behavior, optimize search using tag index * Update plans & pointerplans to support tag serialization * Clean & optimize FSS equality * Add DIF plan version * Add comparison on AST using FSS DIF * Reduce code duplication in FSS * Rm notebooks * Add the 'protocol' keyword to .share * Add support for all bool ops using FSS + add all tests * Fix tests on plans * Fix Protocol to pass tests * Fix baseworker search to handle empty response and empty query * Reduce calls to PRF * Basic FL training plan example * tmp work * Improve plan representation * Fix problems after merge * Tmp work * Add a crypto primitive store * Add option to get the result back of a remote command execution * Add an hybrid version of fake Plans using @tracer * Add fast version with mock plans * Add faster PRG and increase security parameter to 110 * Add comparison support and start xor to add sharing conversion * Add support of fast FFS eq & comp in 1D tensors * Merge conflicts, adapt to a more strict GC process, update tests * Update DPF to support >=1D tensors and add tests for DPF * Add test for EmptyCryptoException + transform crypto_store to a "butter like" store: there is a big tensor and you take part of it depending of the nb of prilmitives that you need * Add support for tensor in DIF class + tests * Add AST comparison using FSS on tensors * rename klass -> class_ * Implement move_ inplace * Update tutorial 9 on worker._objects * Fix tests * Add small caching mecanism * Add comments and clean code * Correct back remote_send * Tiny fix on primitive generation * Address review * Fix th.narrow and comments * Run black * Fix small bugs * Fix bugs Co-authored-by: Vova Manannikov <vmanannikov@broadsoft.com> * Drop `custom` support in SecureNN, Flexibility over dtype in ASTs (OpenMined#3392) * Private method added * Field and torch_dtype for custom tensors * Description for custom dtype * Fix deserialization * Increase sigmoid exp tolerance (OpenMined#3427) * Fix double tracing bug (OpenMined#3428) * fix bug where actions are duplicated when tracing torch functions on Placeholders * remove unused file * Assert `dtype!="custom"` * Assertion msg * Assertion msg * Remove assertion for "core" operations * Apply suggestions from code review Single line comparision Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> * Flexibility over int dtypes * Derp in field -_- * dtype fixed * Drop short and int8 support :) Serde ain't easy peasy Co-authored-by: George-Cristian Muraru <murarugeorgec@gmail.com> Co-authored-by: Jasopaum <jason.paumier@gmail.com> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> * Fix `Plan` tracing that involves `backward()`/autograd (OpenMined#3383) * [WIP] Fix `Plan` tracing that involves `backward()`/autograd * Set `requires_grad` to `True` on `Plan` building temporary args * Initial attempt to autograd Plan tracing * Address review comments * Fix unit tests * Fix gc unit tests * Empty commit Co-authored-by: Vova Manannikov <vova@promail.spb.ru> * Modified securenn to work with >3 parties (OpenMined#3148) * Modified securenn to work with >3 parties * Improved comments and wrap * Cleaned some leftover code * Fixed a bug in maxpool_deriv * Added tests for securenn with >3 parties * Improved usability of maxpool_deriv * run black * Ran black on changed files * Fixed bug * ran black * fixed bug on tests * ran black, again * Move tests from torch to mpc Co-authored-by: Daniel Hurtado Ramírez <trdhrz@gmv.es> Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> * Trace multiple `Protocol Roles` from a single function (OpenMined#3409) * Replace `Protocol` code with `Plan` code (as a starting point) * Remove translation from `Protocol` for now * Set up `func2protocol` on `syft` module * Add a failing `Protocol` test * convert role attribute to a dict of Roles * handle input and output repartition * remove Protocol serde tests * remove pointer_protocol * remove protocol tutorial * remove references to PointerProtocol * remove references to PointerProtocol * fix and add test for Protocol.copy() * comment code still not used * update protocol serde for msgpack * update protocol protobuf serde * Plan and Protocol need to be built before serde * remove input_repartition and output_repartition * remove input/output_repartition from serde tests * remove is_built from serde tests * address review comments * remove last occurrences of state in protocols * remove include_state from tests * update syft-proto dep * remove make_ptr_protocol in serde tests Co-authored-by: Jasopaum <jason.paumier@gmail.com> * Bump version to `v0.2.5` (OpenMined#3451) * Initial part of Auto-Scaling on Google Cloud (OpenMined#3437) * Added autoscale , automatically spin-up instance * Added instructions to set up GCP account * reformatted * fixed the destroy method * moved to grid package * Cast allowed users as Tuple in PrivateTensor (OpenMined#3446) * Allowed users as tuple * Remove pdb * cleanup rnn.py (OpenMined#3441) * cleanup rnn.py * clean up and comment * fix batch_first * Add a check for docstring coverage to the Github Action tests (OpenMined#3462) * Modify Bengali translated notebooks (OpenMined#3415) * add Bengali translated notebook #Part 5. * add translator name. * add Bengali translated notebook #part - 4. * modify bengali translated Part 01 - The Basic Tools of Private Deep Learning notebook. * modify bengali translated Part 02 - Intro to Federated Learning notebook. * modify bengali translated 'Part 06 - Federated Learning on MNIST using a CNN' notebook * modify bengali translated 'Part 06 - Federated Learning on MNIST using a CNN' notebook * modify bengali translated notebook 'Part 07 - Federated Learning with Federated Dataset'. * modify bengali translated notebook Part 08 - Introduction to Plans. * modify bengali translated notebook Part 09 - Intro to Encrypted Programs. * modify bengali translated notebook Part 10: Federated Learning with Encrypted Gradient Aggregation. * modify bengali translated notebook Part 08 bis - Introduction to Protocols. * modify bengali translation Part - 05. * remove Part 08 bis - Introduction to Protocols.ipynb from bengali translation. * modify bengal translation Part 09 - Intro to Encrypted Programs.ipynb notebook. Co-authored-by: Karl Higley <kmhigley@gmail.com> * Update Part 12 Bengali translation- Train an Encrypted Neural Network on Encrypted Data.ipynb (OpenMined#3445) * Update Part 13a Bengali translation - Secure Classification with Syft Keras and TFE - Public Training.ipynb (OpenMined#3449) * Update Part 13b Bengali translation- Secure Classification with Syft Keras and TFE - Secure Model Serving.ipynb (OpenMined#3466) * Store the message log in `BaseWorker` in an unserialized format (OpenMined#3473) * Store the message log in `BaseWorker` in an unserialized format This bypasses some issues where serialization/deserialization changes the state or type of the objects passed to the serdes. That also needs to be fixed, but this is a smaller and still sensible change in the mean time. * Fix comment and formatting * Remove `contents` methods (previously deprecated, partially removed) (OpenMined#3477) * Remove `contents` methods (previously deprecated, partially removed) * Remove stray `contents` check in serde tests * Add async send_command to websocket client (OpenMined#3454) * Add async send_command to websocker client * Update syft/workers/websocket_client.py Co-authored-by: S P Sharan <spsharan2000@gmail.com> * Update base.py * Make print statement only for verbose Co-authored-by: S P Sharan <spsharan2000@gmail.com> * Remove `verbose` flag from code formatting check (OpenMined#3475) The verbose output makes it harder to see where the problems are, because it includes files that already have the correct formatting. That makes it a needle/haystack problem trying to figure out what to change in order for the linting to pass. * Update `Role.register_action()` to allow registering `CommunicationActions` (OpenMined#3465) * Add a test for registering `ComputationActions` in `Roles` * Add a target tensor to the `ComputationAction` test * Add a test for registering a `CommunicationAction` * Add an assertion to `return_ids` to `ComputationAction` test * Update method name in `ComputationAction` test * Update method name in `CommunicationAction` test * Add a check for return_ids in `CommunicationAction` test * Make the `CommunicationAction` registration test pass * Update `CommunicationAction` serialization * Update the surrounding code to use new `CommunicationAction` attrs * Fix the `make_communication_action` serde testing helper w/ new fields * Move shared fields to `Action` abstract base class and update `__init__`s * Move code to produce string representations into `Action` base class * Move `__eq__` into `Action` base class * Move `simplify` and `detail` to abstract `Action` class * Fix remaining issue with msgpack serialization * Move `bufferize` and `unbufferize` to abstract `Action` class * Improve error messages in the `Action` serde test helpers * Add annotations and type checking to `Action.__init__()` to sort out types for serialization * Fix upstream types that feed into `Actions` * Add type annotation to `Action.return_ids` * Take ownership of any objects stored locally * Point `syft-proto` dependency at PR branch * Add a test for taking ownership of objects added to object storage * Add simple `ComputationAction` tests * Remove unnecessary/untested support for lists/dicts as `Action` args * Point `syft-proto` at newly released version * Simplify `un/bufferize_args` by using list comprehensions * Build Roles from func2protocol decorator (OpenMined#3488) * remove owner from placeholder constructor args * interact with roles instead of inputs for Protocols * remove TODO comment * fix serde tests with new syntax * Trace `CommunicationActions` into `Roles` (OpenMined#3448) * Sketch out a comms tracing test * try adding ph.send * Apply `Placeholder` conversion from `handle_func_command` in `send` * Make the `Protocol` test pass again (after rebase) * add more tests 5 pass, 4 fails * formatting * formatting * lint * add test * remove test * get test working * remote_get working * 9 tests working * doc string + clean up * fix empty share test + more doc strings * fix empty share test + more doc strings * fix empty share test + more doc strings * Remove a stray change Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: maddie_shang <mshang@tableau.com> * refactor BaseWorker.send_command to accept explicit arguments. (OpenMined#3487) * syft: refactor BaseWorker.send_command to accept explicit arguments. syft: remove unused variables * syft: fix linter issues. * syft: fix argument type of args_ and update docstring for send_command of BaseWoker class. * syft, pointers: fix int type to tuple conversion. Co-authored-by: Shubham Gupta <shubamgupta3121@gmail.com> * Add reset method on Role (OpenMined#3495) * add reset method on Role * fix reset so that it doesn't erase state placeholders * Plans support for typechecking, lists and dicts using serialized input. (OpenMined#3296) Working on issue OpenMined#3184, improving Plans flexibility. * Adding support for lists, dicts and tuples as input for Plans. * Adding a typecheck when calling a Plan, comparing the call arguments with the built arguments types. * New error when using different length nested structures or input arguments on a Plan at call time, compared to build time. * New error when using dicts at call that have different keys than the dicts with whom the Plan was built. * Added protobuf support for types. * Refactoring the type checking in a new class - NestedTypeWrapper. Co-authored-by: tudor <Tudor Cebere> * Update Part 13c Bengali translation - Secure Classification with Syft Keras and TFE - Private Prediction Client.ipynb (OpenMined#3500) Error fixes in Bengali translation * Move docstring coverage check earlier in test action (OpenMined#3505) This catches insufficient docstring issues up front instead of after 30 minutes of test execution time and cleans up a few of the build steps along the way. * Adding a Protobuf/Msgpack Serializable Interface (OpenMined#3386) * * Changing the logic in the protobuf enroll, now requiring to implement a basic interface (no need to edit syft/serde anymore). * Adapting all the classes to this change. * * Coding style update. * * Cool refactoring * Cleanup. * More cleanup. * * Adding MsgpackInterface and implementing it similar to ProtobufInterface. * Cleanup * * Adding Paillier support in msgpack. * * Reformat * rebasing with master fixes * Changing requirements to the latest syft-proto * Rerunning tests. * Moving files around. Create a single class SyftSerializable, instead of two separate classes. * Adapting all classes to the new interface. Fixing some __init__ issues. Reformatting. * Removing old interface files. * Removing AbstractWorker form serde.py Moved the msgpack import in functions to avoid cyclic dependency. * AbstractWorker fix. * Reformat. * Updating to the last version of syft-proto * Adding docs and adding another global variable check. * Appease the code formatter with whitespace changes * Address `Action` and `Worker` serialization issues after merge * Fix `VirtualWorker` serialization and tests * Fix missing import after resolving conflicts with `master` * Update `syft-proto` version to pull in `VirtualWorker` serialization codes * Updating class to the new interface. * Adding docstrings. Co-authored-by: tudor <Tudor Cebere> Co-authored-by: Karl Higley <kmhigley@gmail.com> * Fix `move` PT to its own location (OpenMined#3442) * Add case: `location == destination` * Add test * Add a missing `NodeClient` import to `AbstractGrid` (OpenMined#3511) This shows up as a `flake` linting error on several PRs after recent changes to the linting rules. * Make an `ObjectStore` something a `Worker` has (instead of is) (OpenMined#3484) * [WIP] Make an `ObjectStore` something a `Worker` has (instead of is) * Update tests to look for objects in `worker.object_store._objects` * Fix references to `_tensors` and `rm_obj()` * Remove `owner` attribute from `ObjectStorage` * Revert "Remove `owner` attribute from `ObjectStorage`" This reverts commit 7d687fb. * Fix loose ends with setting `owner` attribute * Fix `ObjectStorage` tests * Fix `test_share_get` * Fix the Udacity backwards compatibility tests * Clean up `AbstractObject.tag()` * Fix `get_obj` references in `FederatedClient` * Replace direct access to `_objects` with `get_obj()` * Fix tag registration * Remove stray comment * Use `clear_objects()` method instead of setting to `{}` * Add a missing `NodeClient` import to `AbstractGrid` * Rename `ObjectStorage` to `ObjectStore` * Tone down triple exclamation points in error messages * Remove unnecessary guard clause and exception from `Object.tag()` * Clean up and expand the Udacity course compatibility comments * Add `get_obj` to `FederatedClient` * Updated PR Template (OpenMined#3506) * Updated PR Template * removed checks * removed checklist item 'add test' * More frameworks when tracing (OpenMined#3461) * More frameworks when tracing * Update syft/execution/plan.py Co-authored-by: Karl Higley <kmhigley@gmail.com> * Register frameworks * Move comment * Remove trace * Add equal for gaussian mixture * Add NodeClient Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: systemshift <42102034+systemshift@users.noreply.github.com> Co-authored-by: Jasopaum <jason.paumier@gmail.com> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> Co-authored-by: Vova Manannikov <vmanannikov@broadsoft.com> Co-authored-by: S P Sharan <spsharan2000@gmail.com> Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: Vova Manannikov <vova@promail.spb.ru> Co-authored-by: knexator <daniel.hurtadoramirez314@gmail.com> Co-authored-by: Daniel Hurtado Ramírez <trdhrz@gmv.es> Co-authored-by: Anshuman Singh <anz19999999999@gmail.com> Co-authored-by: David <2524133+dcato98@users.noreply.github.com> Co-authored-by: jabertuhin <jabertuhin@gmail.com> Co-authored-by: Zarreen Naowal Reza <zarreen.naowal.reza@gmail.com> Co-authored-by: Madeleine Shang <shang.madeleine@gmail.com> Co-authored-by: maddie_shang <mshang@tableau.com> Co-authored-by: Shubham Gupta <shubhamgupta3121@gmail.com> Co-authored-by: Shubham Gupta <shubamgupta3121@gmail.com> Co-authored-by: Tudor Cebere <tudorcebere@gmail.com> Co-authored-by: Utkarsh Srivastava <39618393+tkrsh@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ure Model Serving.ipynb
Description
Thank you for your contribution to the PySyft repository.
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
I have reviewed the bengali translation of this tutorial and fixed some major bugs and incoherence in language.
If your pull request closes a GitHub issue, then set its number below.
Fixes # (issue)
Type of change
Please mark the options that are relevant.
Checklist: