This repository has been archived by the owner on Feb 16, 2023. It is now read-only.
Update PyGrid tutorials #528
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@hericlesme I did some changes, could you review it again, please? :) |
|
@IonesioJunior LGTM. Just a question: are we going to put or rewrite the tutorials somewhere? |
IonesioJunior
approved these changes
Mar 31, 2020
IonesioJunior
added a commit
that referenced
this pull request
May 12, 2020
* pip-dep folder & requirements_dev.txt * reqs moved to pip-dep folder * Makefiled updated to pip-dep folder * reqs moved to pip-dep folder * git action updated to pip-dep folder * Update run-tests.yml * initial toy websocket based example * removed pycharm files * added support files * Added example client/worker * renamed classes * intermediate commit * file running * initial grid support * added worker discovery to flask server * changed session for sqlalchemy datbase * added known workers to GridClient * checkpoint * initialized grid client and app * clean dev * reinit * a few edits * Added destroy() * added notebook * Added warning * updated warning * updated warning * updated warning * updated warning * Link PySyft in README file (#193) * small bugfix * Fixing heroku check (#194) * fixing heroku check * fixing version check for heroku * syft update * trask_grid -> dev * Fix #196 (#198) * Clean up (#201) * Clean up app.py * Create utils and clean up init * Clean up client * Clean up: part 2 (#212) * Clean up app.py * Create utils and clean up init * Clean up client * Fix typo * Create dev_user and force params * Add verify identity * Fix local redis url * cd to grid folder before checkout dev * Fix verify identity * Adding tests and how to execute them * Rename: tests -> test * Remove force (not working as expected) * Delete test_local_grid_client.py * Moving SocketIO Client and Server Worker from PySyft to Grid (#218) * fixing broken test * moving socketIO workers to grid project * Move socketio (#219) * fixing broken test * moving socketIO workers to grid project * fixing tests * Pre-commit, tests, contributors guideline (#220) * Pre-commit, tests, contributors guideline * PySyft -> Grid * Add Makefile * Examples to run with Android app (#221) * Examples to run with Android app * Fix imports * Add grid websocket approach (#217) * ADD async websocket grid app * ADD sync websocket client class * Update requirements/ change app * Rename App * Change /identity response * Deployment of websocket application * Unify address parameters at WebsocketGridClient * Little change at Procfile to deploy app with flask socket worker config * Update websocket app requirements.txt * Add -k flag at Procfile * Add eventlet worker to manage network app concurrency * [Refactor] Rename flask dir to rest_api * Add WebsocketGridClient class at grid module * ADD binascii operations * Change websocket application * Update requirements.txt * Add jupyter notebook example * Remove var type at docstrings * Create LICENSE * Add python-socketio at requirements list (#231) * Run Black (#233) * Run Black * Add coverage to requirements * Add syft to requirements * Add binder, build status, slack and FOSSA tags (#234) * Update requirements.txt (#237) * Tensor persist (#235) * fixing bug where localhost redis path not good * wip for getting postgres with a schema * moving socketIO workers to grid project * started getting flask integration test working and postgres migration * adding migration * fixing broken test * moving socketIO workers to grid project * fixing tests * got worker -> worker_objects relation working * removing whitespace * removing dead code * format fix * adding flask_sqlalchemy * adding flask_migrate * removing redundant import * unnecessary for now * commented for now * cleaning up imports * removing dead code * cleaning up more imports * added a description * Fix #238 (#242) * Grid communication (#245) * Add communication between grid nodes * Add grid at requirements.txt * Set is_client_worker flag to false / remove auto verbose mode * Run black format tool * Fix websocket app grid dependency at requirements.txt * Change log/debug message at connect grid node callback * Remove know_nodes dict * Performance Optimization between Grid nodes communication (#249) * Performance Optmization * Add notebook to measure performance before/after remove sleep procedure * Remove duplicated notebook * Add reference to Pysyft tutorial * added check for windows for tmp folder deletion (#250) * added check for windows so that tmp folder is deleted only for non-windows platform in case grid already exists * added check for verbose to output command execution status * Add ConnectionError Exception (#252) * Add ConnectionError Exception at WebsocketGrid Client * Fix code style pattern * removed flake8 docstrings from pytest in travis so travis could run (#257) * Add architecture diagram (#254) * Add architecture diagram * Improvements to the diagram * Fix notebook after changes in serde (#255) * rename tutorial file for windows compatibility (#261) Can't checkout file on windows because of ":" in the filename. Also, to be consistent with pysyft tutorials naming style: https://github.com/OpenMined/PySyft/tree/dev/examples/tutorials * REFACTORY Grid Node Application (#264) * Heroku script for creating app instance (#263) * Tensor persist router (#248) * fixing bug where localhost redis path not good * wip for getting postgres with a schema * moving socketIO workers to grid project * started getting flask integration test working and postgres migration * adding migration * fixing broken test * moving socketIO workers to grid project * fixing tests * got worker -> worker_objects relation working * removing whitespace * removing dead code * format fix * adding flask_sqlalchemy * adding flask_migrate * removing redundant import * unnecessary for now * commented for now * cleaning up imports * removing dead code * cleaning up more imports * added a description * starting flask api tests * adding persistence support for grid with unit tests * adding notes * removing unnecessary imports * removing hidden file * ignoring .DS_Store * formatting * renaming test * renaming test * better assertions * adding verbose mode * adding a workbook * green tests * reformatting * changing heroku script to use postgres * adding Flask-Migrate dependency * adding requirements * moving postgres rest api into the app folder * moving create_app * heroku app * reqs * adjusting module imports so that tests and heroku script will both work * fix import * back again * adding migration command * formatting * fix test coverage * formatting * Grid gateway (#267) * Create a initial version of grid gateway component * ADD remotely search method * ADD FederatedClient inheritance at GridNodeClient * ADD node/gateway dockerfiles + docker-compose config * ADD Search / Join Gateway routes * ADD Grid Network class * ADD /search route at grid node * ADD sy.Message() at search method * FIX Code Style * FIX route conflict * REMOVE gateway unused dependencies * FIX bug at GridNetwork search method * UPDATE README.md * UPDATE README.md * ADD new notebooks * FIX merge review requests * Change to Grid@dev at requirements.txt * run black tool * Update websocket_client.py * Update websocket_client.py * FIX REST_API_TEST (Update to sy.Message structure) * FIX connect_grid_node BUG / Run black tool * ADD WebsocketClient/Grid tests * ADD flask_cors package at requirements file * ADD eventlet package at requirements * ADD docker documentation reference at README.md file * FIX time interval at websocket/grid tests * UPDATE README.md / Decrease time interval between connections * FIX GridNetwork connection bug * REFACTORY tests * run black tool * REMOVE SocketIOAPITest class (change all test cases to pytest design) * REMOVE time.sleep() from socketio test module * run black tool * REMOVE stress test * Revert "REMOVE stress test" This reverts commit 5e330fb051dd7f4845d0f5ab5ca4baf25f3cbb3b. * Remove parameters from connect_nodes test * FIX test_connect_nodes * Update requirement files - Fix a stable version to syft lib - Remove redundant dependencies - Add requests lib at gateway requirements * Change Dockerfile's base image - python-slim base image is more lighter than ubuntu image * ADD new environment variable at gateway compose configs - Now, the gateway component will receive the secret key by environment variable * Revert changes at requirements.txt (websocket_app) - Download syft/torch packages directly from pip spend much disk space. - We have limited space on heroku platform (500 MB) * [FIX] Problem to finish grid node processes after finish test cases * run black tool * ADD time interval between websocket tests - Limited hardware resources to test environment * ADD time interval between node connections * Review requests * Deploy module (#273) * ADD deploy module - ADD BaseDeployment class (abstrac class used to initialize generic attributes) - ADD NodeDeployment class (class used to perform grid node applications deployment) * Update "Launch Websocket Grid Node" notebook - Now, perform grid node app deployment with deploy module * FIX REST_API_TEST (Update to sy.Message structure) * Sync with syft repository - In the last syft commit, the AbstractTensor class was moved to generic module. * Update requirements.txt list - Install the last version of syft library. * UPDATE requirements.txt - Add egg-name to syft package * Revert "Sync with syft repository" - This reverts commit d4c8b4f777b06438e6740392cc901f6b75fb72f0. - These changes have not yet been applied in the syft release version. * Revert changes on requirements.txt * Review requests * RENAME o -> output * FIX undefined variable name * FIX wrong var name * FIX undefined var * RENAME component -> base_deployment source file * Check if gateway_url is not None before using it + fix tests (#276) * Check if gateway_url is not None before using it * import AbstractTensor from syft.generic.tensor * Update package imports (#275) * Updated README.md (#278) * Update README.md With dev branch tutorials and projects page. * Update README.md * Launch on Heroku Update (#280) * __init__.py: - Fixed a typo in execute_command(command) - Added documentation to launch_on_heroku function - Removed unessacery if from launch_on_heroku utils.py: - Added documentation * Updated Part1 example notebook * Update __init__.py * Update utils.py * ran black for formatting on grid * Grid Node Persistence (#277) * ADD SQLAlchemy to grid websocket application - Adapt Jon's persistence features to websocket context. * Check if gateway_url is not None before using it + fix tests (#276) * Check if gateway_url is not None before using it * import AbstractTensor from syft.generic.tensor * Update package imports (#275) * Implement persistence with SQL Alchemy - Refactory SQLAlchemy Models - Remove set ID websocket event - Add recover_worker / snapshot methods * Revert set-grid-id endpoint - To the future, we'll remove this endpoint (This is not the purpose of this pull request). - Each grid node must initialize with its own id, an external component cannot change it. * MOVE db module to main package * run black tool * Snapshot method performance optmization - Before, for every command, we deleted ALL items registered on database and added ALL items again. - Now, for every command, we add/remove the difference between current and last snapshot. * RENAME db_module -> persistence * ADD doc comments * Updated README.md (#278) * Update README.md With dev branch tutorials and projects page. * Update README.md * FIX review requests * Refactor tests (#283) * IDs are lowercase and use a less common gateway port * Create folders * Clean up grid search tests * Update docker environment (#286) * Fix #271 and #287 (#285) * Fix #271 * Increase sleep time * Undo changes in test_api_socketio * Rename test * decrease sleep time * Add test for connect_all_node and fix #287 * Host model on grid node (#272) * Create port argument for websocket_app * Websocket Client inherits from Client * Create serve model route * Create experimental notebook * Return proper response * Create route for model prediction * Create _send_get * Create run_inference function * Rename model_name -> model_id and list models * Move model related methods to client * Update notebook * Clean up warnings * Create unhexlify argument * Increase ping timeout and ping interval * Support plans and jit models * Delete 2. Host a model on a grid node.ipynb * Delete Host a model on a grid node.ipynb * Organize examples folder * Improve tutorials * Move tutorials from experimental * Clean up encoding * Re-run notebook * Add tests * Update client.py * ADD support to models on Grid Gateway (#291) * Increase Grid Network support to models - ADD /choose-model-host route at gateway - ADD /search-model route at gateway - ADD /search-available-models route at gateway - ADD /search-available-tags route at gateway - ADD /dataset-tags route at websocket app - Create a new notebook * ADD unit tests (host jit model / query_model) * Improve gateway's index page * Add random.sample at choose hosts endpoint * Return the first match_node * Revert "Improve gateway's index page" This reverts commit 83af10108bcdb8e50d9a4684d9a04d87faaad734. * Solve review requests - Change list objects to set objects - Fix typo - Set ConnectionError Exception explicitly - Move n_replica's var to gateway.py * ADD host/query plans test * run black tool * Solve review requests - Remove if at gateway.py - ADD unit tests to /search-available-tags, /search-available-models endpoints * ADD run_inference method at grid - ADD run_inference method - Update Notebook - Update tests * Change host_model -> serve_model * Update Notebook * UPDATE grid tests * HOST GPT-2 (#292) * ADD new Notebook * Fix Notebook * Fix typo * ADD requests-toolbelt dependencies * ADD Text generation at Notebook * Solve review requests * Update requirements.txt (#294) - Fixes deployment issues on Heroku with size - Added Postgres driver * Host and Query an Encrypted model (#288) * Create 3. Host and Query an Encrypted model * Rename 3. Host and Query an Encrypted model to 3. Host and Query an Encrypted model.ipynb * Change model to convnet * Use a plan * Support fetching a plan * Remove unused notebook cells * Restart notebook to show that fetch is working properly * Use convnet * Get pointers instead of getting objects * Update for changes in fetch * Add id to model * Support private model hosting and querying * Create a new notebook * Fix notebook * Update reference to correct attribute * Clean up and add comments * Unify serve model and document get_private_model * Add persistent models (#282) * Create port argument for websocket_app * Websocket Client inherits from Client * Create serve model route * Create experimental notebook * Return proper response * Create route for model prediction * Create _send_get * Create run_inference function * Rename model_name -> model_id and list models * Move model related methods to client * Update notebook * Clean up warnings * Create unhexlify argument * Increase ping timeout and ping interval * Persistant Model Storage - model_manager.py for saving, deleting, listing and in short handling all the models - routes.py: updated for model manager and created a new route of deleting a model - jupyter notebook example updated with the delete function and some commenting to make it easy to understand - client.py updated with the delete_model route * A sample pickle model save file. * Support plans and jit models * Delete 2. Host a model on a grid node.ipynb * Delete Host a model on a grid node.ipynb * Organize examples folder * - Added database * Improve tutorials * Move tutorials from experimental * Clean up encoding * Re-run notebook * Add tests * deleted model files. As they are shifted to db. * Improved Comments - Improved comments - Removed conflicts - Improved exception handling * model cache - removed model list from routes - Adding model cache to model_manager. * removed todo * ran black. * Style Fixes - Style fixes - model_manager now returns dicts - Updated client to handle new requests - Success status added in model manager and flask responses. * Merged with dev - Merged with dev - Ran black - Removed unnecessary prints * fixed bug in check_if_model_exists - fixed bug in check_if_model_exists. - better comments and return dicts. * Use LargeBinary instead of BYTEA * Rename MLModel -> TorchModel * Style fixes and fix clear cache * Add missing json loads * Add delete operation * Remove experimental notebook * Add missing json.loads * Remove unused variable and add comment * Add TorchTensor * Create local_worker var * Create file with local worker utilities * Cache real object + fix plans db support * Use local worker utilities * Add tests for deleting a model * Use encoding * Fix GPT-2 demo * Rename fetch_plan to fetch_plan_reference * Fix imports * Use sy.messaging * Fix Message * Fix imports * Create notebook 2.2 * Delete 3. Host and Query an Encrypted model.ipynb * Avoid error during snapshot * Host and Query encrypted models options * Remove fetch_plan_reference and demo 3.3 * Remove get_private_model and demo 3.1 * Rename notebook * Fix make_helper jit support * Add db support for encrypted models * Use local_worker utilities * Clean up model manager * Relaocate comments * Fix issues in Syft for now * Fix torchscript db issue * Add test test_host_encrypted_model * Fix test_host_encrypted_model * Replace 2.2 notebook * Add test_run_encrypted_model * Clean up websocket app and gateway app starting scripts (#296) * Fix typo (tst_config -> test_config) * Use parser and os environ vars as default * Fix import order and typos * Define warning * Fix tests * Update requirements.txt (#297) - fetching syft from dev branch directly. * Clean up host model API (#298) * Add new columns to TorchModel * Add allow_model_copy and allow_run_inference args * Snapshot shouldn't include model objects * Fix tests * Create get_serialized_model_with_id * Create get_model route * Create get_model_copy * Create global variables for storing error messages * Rename allow_model_copy -> allow_get_model_copy * Fix typo * Implement serve_encrypted_model * Improve comments * Add tests for new arguments * Fix test * Standardize results * Update tests * Fix small bug * Improve delete error messages * Fix typo * Fix test * Extend syft with a friendly api * Better argument names * Try to fix tests * Try to fix tests * Fix tests * Added TODO * Update notebooks * download -> download_model * Wrap host/inference encrypted models (#300) * Wrap host/inference encrypted models - Wrap encrypted serve_model / run_inference - ADD /search-encrypted-models at websocket app - ADD /search-encrypted-model / /choose-encrypted-host at grid gateway * Fix typo * ADD encrypted serve_model/run_inference test * FIX serve_model/run_inference test * Fix typo / ADD constant value * Fix review requests * Update requirements.txt (#297) - fetching syft from dev branch directly. * Run black tool * ADD comment about eventlet money_patch * ADD new notebook * Fix heroku ws app (#302) * Improve 3.2 * Fix heroku app * Get rid of main * Update notebook * Fix websocket_app.py * args.gateway_url -> gateway_url * Change Model ids (db.Integer -> db.BigInteger) * Update notebook * Remove outdated comment * Add Introduction Notebooks (#303) * Start What's Grid Notebook * Update and rename 1 - What's Grid?.ipynb to Part 0 - Introduction to Grid * Create Part 1 - Launch a Grid Node Locally.ipynb * Update and rename Part 1 - Launch a Grid Node on Heroku.ipynb to Part 2 - Launch a Grid Node on Heroku.ipynb * Update Notebook (Launch a Grid Node on Heroku) * ADD .ipynb extension at notebook * FIX Launch a Grid Node on Heroku notebook * ADD deployment support to grid gateway * Fix Procfile / gateway.py * Fix websocket_app bug * ADD heroku_gateway deployment module * Improve notebooks * Update demo (#305) * Improve 3.2 * Update demos * Update notebooks * Add note about checking tutorials * Remove Serving models * Review examples (#307) * improve readability of tutorials (#308) * Fix tutorial serving querying (#310) * Fix for psycopg2 package installation on Linux * Fixed torch 1.0.1 requirement for Linux and Windows * Fixed imports' names for the gp2_utils file and added explanation on how to correctly parse the HMNIST dataset * Reverted changes related to dependencies for torch * Reverted psycopg fix * small improvements in readme and tutorial (#313) * small improvements in readme and tutorial * correcting typos and adding some explanations * Fix typos in tutorials (#309) * Fix typos * Update Part 1 - Launch a Grid Node Locally.ipynb * Fix typo * Fix test generation * General fixes / improvements * ADD Gateway persistence (#314) * ADD Gateway persistence * ADD Persistence at Gateway Deployment * ADD Notebook * Update 4. Introducing the Grid Network.ipynb (#316) * Web Index page update (#317) * web initial * model list being fetched from server * finalized web index page * ran black * changed gradient background - to cover. * added comment for model.pop explanation. * fixed typo - removed test endpoint. * ran black * ADD webpage section (#322) * Fix for psycopg2 and torch 1.0.1 package installation on Linux (#301) * Fix for psycopg2 package installation on Linux * Fixed torch 1.0.1 requirement for Linux and Windows * Fixed PyTorch package version for Python 3.7 * Added python runtime environment for Heroku * Disallow cross-origin resource sharing to webpage routes (#323) * Add README files for app folders (#324) * Improve 3.2 * Add README.md files in app * Add grid logo to Readme (#329) * Update README.md added logo and changed grid to PyGrid. * Update README.md * Update model_manager.py (#331) fixes #318 - in case where save_model gives an integrity error * Common Install Issues Section (#334) * Common Installatoin Issues Section fixes #306 * Update README.md * Add api documentation (#328) * Verify download permissions at /get_model/<model_id> route - If someone performs a request to this endpoint outside of grid lib context, they probably will get the model (Security Problem). - Now, /get_model/<model_id> endpoint returns 403 (Forbidden) if not allowed to download. - Now, /get_model/<model_id> endpoint returns 404 (Not Found) If not found at the grid node. * CHANGE /model/<model_id> request method GET -> POST We can forward HTTP body content at GET requests, but it's a bad practice. We should use POST/PUT methods if we want to forward important data between REST API. * Change response error code at /serve-model - Replace 500 Internal Server Error -> 409 Conflict 5xx rest codes are used to represent server errors. 4xx rest codes are used to represent client errors. In this case, our response error message will be sended when we have a already registered model This error was be created by the client side, not server. * ADD Grid Node REST API documentation * Fix markdown * Fix grid_node_rest_api document * ADD Grid Node Websocket API Documentation * ADD gateway REST API * FIX gateway persistence bug * FIX Notebooks examples * FIX api documents * FIX gateway_api document * Merge grid clients (#333) * Merge WebsocketGridClient and GridClient - Move all GridClient's methods to WebsocketGridClient * Remove GridClient class - Remove GridClient reference. - Remove remove GridClient tests. * FIX flake error * Merge Grid Node Apps (#335) * Remove unused apps * Remove outdated/unused tests * Remove notebook(copy) * Socket Grid Node (#338) * [REFACTORY] Grid Node App - Change SocketIO protocol to Pure Websockets Protocol (Now we can send/receive messages from syft.js) - Move some HTTP routes to socket API - Allow to receive/send binary contents (Perfomance Improvement) - Add event_routes source file (handle socket requests) * [REFACTORY] Grid Lib - Remove unused/outdated source file (grid/cliemt.py) - Extends syft.WebsocketClientWorker - Change HTTP node requests to socket node requests - Delete unused/outdated methods * [CHANGES] Tests - Change conftest (update to test new grid_nodes) - Rename test_api_socketio -> test_api_socket - test_api_sockets changes (update to test new grid_nodes) - test_api_grid changes (update to test new grid lib) * ADD gevent/gevent-websocket at requirements.txt * Updaate Procfile * Update requirements dependencies * Fix for psycopg2 on gateway (#344) Replaced the `psycopg2` package with `psycopg2-binary` in the gateway/requirements.txt file. * Update travis badge (#346) * Update travis badge Replaced `https://travis-ci.org/OpenMined/Grid` to `https://travis-ci.org/OpenMined/PyGrid` * Skip test_grid_search Add @pytest.mark.skip decorator at test_grid_search unit test * fixing formatting error * ordered requirements to install grid (#347) * Update the PyGrid components web user interface (#348) * update titles from 'Grid' to 'PyGrid' in web interface * create web interface for pygrid network based on node interface * Add Issue Templates (#349) * create issue template: feature request * create issue template: bug report * Add sphinx config to build docs (#351) * Add sphinx config to build docs Edit conf.py to set rtcat theme. Set napoleon to allow Google style pydocs. * Add documentation build to travis CI verifications (#353) * add new parameter to docker-compose (#340) * add dataset tags table and fix ui grid (#357) * Update index.html (#358) * Update template.css (#361) * Add setup.cfg/pbr and make package Ready-to-Publish (#362) * Add pbr and setup.cfg * Edit setup.cfg and README for valid pypi package Edit invalid setup fields. Set README image to hosted files at Github. * Add Pull Request template (#364) * Add Pull Request template * Edit PR template to clarify requests * Edit PR template to remove typo * Update PyGrid-Arch Image (#365) * Add PyGrid-Arch image * Update PyGrid Architecture Image on README * Authentication (#342) * ADD Auth module at grid lib * Update grid __init__.py * ADD Auth module at Grid Node App - Create Session Repository - Create UserSession - Socket Session Auth wrapper - Authenticate socket route * ADD authenticate method * SKIP test_host_plans * Fix problem with _load_preset_credentials * ADD flask_login lib * Handle Multiple Private/Public Workers - Now we're able to connect using different private workers at the same node. - Map private workers by user attestation. - We're able to connect with local private/public workers. - We're able connect private/public workers with remote nodes. * Remove unused source file * ADD docstrings * Remove custom search method at grid node * Remove search method at grid.websocket_client * Remove unused imports * Update Authenticate event response * Move model_manager source file to persistence module * Fix /workers/ endpoint * ADD default admin infos * Update auth module * Fix search method * RENAME create_standard_auth -> register_new_credential * RENAME load_auth_credentials -> load_credentials * ADD docstrings at auth module * MOVE 'set_database_config' to persistence module * REMOVE local_worker_utils source file * ADD docstrings at PyGrid App * REMOVE outdated functions 'launch_on_heroku' * FIX/ADD Docstrings at PyGrid lib * CREATE grid_codes source files * REFACTORY events - Create events module at PyGrid Node App - events.py -> events/__init__.py - event_routes -> syft_events.py, model_events.py, control_events.py * Update grid_codes.py * Skip Plans CI tests - PySyft last update crash Plans CI tests/ grid functions, it will fixed in another PR. * Fix flake/black code style errors * Add entrypoint.sh (#368) * route to delete a grid node (#367) * route to delete a grid node * fixing routes with black * runing black at manager file * Bump werkzeug from 0.15.2 to 0.15.3 in /gateway (#369) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.15.2 to 0.15.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/0.15.2...0.15.3) Signed-off-by: dependabot[bot] <support@github.com> * Adding MNIST Federated Example (#366) * Adding MNIST Federated Example * Cleaning notebooks outputs/Adding cuda support to mnist federated example * Adding empty folder to download datasets / Changing datasets path * Updating files permissions * Bump werkzeug from 0.15.2 to 0.15.3 in /app/websocket (#370) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.15.2 to 0.15.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/0.15.2...0.15.3) Signed-off-by: dependabot[bot] <support@github.com> * Edit docs to serve it remotely (#375) * Mock deps for autodoc * Replace autodoc for autoapi Set autoapi to averts heavy dependencies * Edit sphinx conf to enforce theme css * Edit sphinx to include generated api documentation * Conform code to black standards * Fix file permissions (#376) * updating websocket route event (#378) * Update local examples to conform to grid module (#382) * Fix serde and logging imports in Android worker example notebook (#380) * Move documentation/ files into docs hierarchy (#384) * Updated torch version for worker container (#386) * Replace Grid with PyGrid (#387) * Add spam Example (#388) * Add files from SPAM Folder * Spam Populate Grid Network Notebook * Spam Train Model * Black reformat for Spam example * Spam fix typo (#391) * Fix bug on remote inference (#392) * updating websocket route event * undo commit * fixin response message on websocket app * Start Android example server on `0.0.0.0` to accept all connections (#394) Change address to '0.0.0.0' to accept all connections * Download model bug (#396) * updating websocket route event * undo commit * changing way to get serialized model * Change PySyft branch from dev -> master (#397) PySyft no longer hosts its latest code in the dev branch. We push straight to master. * Remove unused/outdated import (#398) * Refactory gateway Routes (#403) * Gateway Routes Refactory - Decrease Cyclomatic complexity. - Avoid/Handle Internal Server Errors * ADD INVALID_JSON_FORMAT_MESSAGE constant * Fix the websocket app procfile module to websocket_app (#404) * Rename folders used in heroku deploy scripts from grid to PyGrid (#406) * Add missing proto dependency to the websocket app (#405) * Process tensor request (#399) * Remove unused/outdated import * Handle GetNotPermittedError and save tensor request * Add CI tests * Update PySyft repository reference * Fix requirements.txt * Fix .travis install session * Add egg information to requirements.txt (#409) On my machine (OSX 10.14.6 running python 3.6), while I was able to install requirements.txt using 'pip install -r requirements.txt', I was not ablee to do so by running 'pip install .' as the README directs. After some googling, adding egg information to the github imports in requirements.txt allowed me to do so. I don't claim to be an expert on this issue, but it worked for me. * remove_Connecting_Locally.ipynb (#414) * Decode error (#400) * updating websocket route event * undo commit * adding try catch to decode bytes that utf-8 cant decode * [FIX] serve_models / run_encrypted_inferences on plans. (#420) * [GridNode] Fix host/run_inference on plans * [PyGrid Lib] Update serve_model method. * [CI Tests] Allow / Update tests using plans. * Fix encrypted inference route * [CI Tests] Allow/Update host/run_inference/run_encrypted_inferences at grid tests. * Fix requirements.txt for heroku The syft_proto is currently failing to install on Heroku with the HerokuNodeDeployment script. Trying to fix it * Use PyPI version of syft_proto * Bugfix * Bugfix * Fix app requirements.txt (#422) * [Bug FIX] Standard tensors also have the get method, even without children. (#425) * Fix requirements hierarchy (#423) * Fix app requirements.txt * Remove proto dependency from requirements.txt, fix dependency hierarchy (it'll avoid package incompatibilities between syft/pygrid and syft-proto) * Update docker-compose.yml (#424) * Update docke-compose / README * Fix README.md * Correct Fell->Feel type in the README. (#426) There is a noticeable typo. Please accept this patch to fix it. * Kubernetes Config file (#415) * docker files * Updated commands formatting * Added services, removed unnecessary entries from deployment * connecting to the local gateway service * moved files to appropriate base folders * rationalised the label names, updated image of node to default * add gateway overlays for dev and prod. * basic kustomize working for gateway * started using config maps for env vars * used configmaps for node * deploy script * updated service to be available externally * Fixed the deploy to proper shape * added todo, renamed k8s to readme * updated readme.ms * Update todo.md * added docker build grid-gateway, grid-node * removed changes for to travis.yml * update docker image * support for multiple nodes * support for multiple nodes in prod * support for multiple nodes in test Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Updated k8s Readme (#427) * docker files * Updated commands formatting * Added services, removed unnecessary entries from deployment * connecting to the local gateway service * moved files to appropriate base folders * rationalised the label names, updated image of node to default * add gateway overlays for dev and prod. * basic kustomize working for gateway * started using config maps for env vars * used configmaps for node * deploy script * updated service to be available externally * Fixed the deploy to proper shape * added todo, renamed k8s to readme * updated readme.ms * Update todo.md * added docker build grid-gateway, grid-node * removed changes for to travis.yml * update docker image * support for multiple nodes * support for multiple nodes in prod * support for multiple nodes in test * Updated the docker build image name * updated Readme.md with kubernetes, docker image name Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Add mpc flag / column for encrypted models (#428) * Replacing Grid Node's PyGrid/PySyft dependencies (#429) * REFACTORY: replace lib (grid -> syft) at grid node app. * REFACTORY: replace lib (grid -> syft) at CI tests. * Update syft temporarily * remove grid imports * ADD private_grid unit tests * Update requirement files * Remove unused imports * Add import warning Co-authored-by: Andrew Trask <liamtrask@gmail.com> * [Refactory] Grid node's persistence module (#432) * Remove SQLAlchemy models / source files * Refactory Persistence module * Fix syft remote search * Update websocketapp / syft_events - Update websocketapp to receive redis url by args / environment var - Update syft_events to import object_storage and use it to recover worker * Fix get model from cache * Remove model instance checkup * Remove save/get plan states * Update model events * Fix/Update persistence module * Update CI tests configs * Update gitignore / add database source file * Run black tool * Update requirements.txt * Update requirements.txt * Change event error message * storage cache encapsulation * Fix/Update routes * Update database server * Replace literal strings by codes * Replace literal strings by syft.codes * Add docstrings * Updating grid status and name at grid app (#443) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * Grid Standardization (#447) * starting gateway implementation over websockets * create ws blueprint * fix some imports * Fix flake/black issues * Add ws events sketch * - Create Scope Class - Create Scope Controller - implement get-protocol message * create codes for response * Add socket handler to manage connections * Fix get-protocol endpoint glitches * Implement webrtc routes * fix black errors * Add docstrings * create test file for gateway websockets api * add socket client for test mocking * update gateway app runner on conftest and fix socket ping endpoint * update requirements.txt with aiounittest * add socket_ping test * add some tests for gateway websockets * fix black issues Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Setup GitHub Actions Workflow (#448) * setup github actions workflow * exclude syft * exclude syft * update syft repo * skip some tests and update syft origin * remove travis and update build status badge (#449) * Consistent Model Hosts (#450) * Search nodes for ones already hosting the model when choosing model host * Add a test for model host consistency * Improve k8s documentation (#452) * Update requirements.txt (#442) Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Create Federated Learning Processes (#458) * create a draft of a federated learning process class * create a controller to manage federated learning processes * fix black issues * update classname and make client_protocols a default arg * Use docker port binding instead of host networking (#461) Solves #460 * working docker compose after removing extra '/' in REDISCLOUD_URL. Both notebooks https://github.com/OpenMined/PySyft/blob/master/examples/tutorials/grid/federated_learning/mnist/ should works now Need to add 127.0.0.1 -> host.docker.internal to /etc/hosts to work around Docker for Mac not supporting network_mode: 'host' (#462) Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Create Workers (#470) * create possible user class * create worker concept * WIP: fl_download_model_endpoint (#471) * Change PySyft Reference (#474) * Add flask ws worker as a Docker ENTRYPOINT (#475) * Remove outdated grid library source code (#476) * Remove outdated grid lib * Remove library documentation * remove unused file * Remove grid lib config files * Update Git CI tests * Remove outdated tests using grid library * Update README.md (#477) * Update README * Add PyGrid image * Remove image from README.md * remove readthedocs file * Updating Launch examples (#478) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * adding route to download protocol * getting the issue done * fixing merge to PyGrid dev * download protocol http endpoint * download protocol http endpoint * download protocol http endpoint * finishing issue #446 * deleting federated route * updating examples how launch grid * changing python version * changing python version * changing python version part3 * changing python version part3 * changing python version part4 Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Sketch Events for Mobile/Web Federated Learning (#469) * Refactory pygrid codes * Remove scope events * Add fl_events sketch * Update events module * Skip scope tests * Fix code style * Remove unused codes * Fix expected JSON fields * Add model_id / remaining_time * Fix code style * Add FederatedLearningCycle class * Fix imports * Minor fixes * Fix Flake8 issues * Fix black tool issues * Support Cycle sequences * Rename FLPController -> FLController * Fix import FLPController -> FLController * Protocol download endpoint (#480) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * adding route to download protocol * getting the issue done * fixing merge to PyGrid dev * download protocol http endpoint * download protocol http endpoint * download protocol http endpoint * finishing issue #446 * deleting federated route * updating examples how launch grid * changing python version * changing python version * changing python version part3 * changing python version part3 * changing python version part4 * adding new route to get protocols * changing branch * fixing black * changing protocols to cliente_protocols Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Setup GH Actions Workflow for Automated Docker build (#484) * Create docker-build.yml * Update run-tests.yml * Delete docker-build.yml * Create docker-build.yml * Create docker-build.yml * Update docker-build.yml * Update docker-build.yml * Update docker-build.yml * remove conditional * update grid node path * Update CI badges in README (#485) * Create ansible playbook to deploy the pygrid (#472) * Create ansible playbook to deploy the pygrid * Refactor playbook/README and add destroyer playbook * update README with destroy playbook usage * remove unecessary vars reference Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Update README and Contributing guidelines (#488) * Update links and numbers * Update CONTRIBUTING.md * Update requirements.txt (#492) * Update Docker Publish Workflow (#487) * Update docker-build.yml * Rename Job Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fl report endpoint (#489) * report endpoint Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * pr for feedback fl_impFLcycles - use server config (#479) * WIP: fl_download_model_endpoint * mock up use of server_config params for #438-2 * mock up use of server_config params for #438-2 * math stuff * wip: model in place * wip: model in place * wip: model in place * wip: model in place * wip: model in place * comment code * comment code * comment code Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Add FL http cycle request endpoint (#497) * [WIP] add federated/cycle-request endpoint to gateway routes * [WIP] enable worker to get cycles and update cycle-request endpoint * no changes. just to avoid overwritten by merge * (fix) remove get_cycles function * PyGrid Gateway Persistence (#481) * Initial commit * Fix tablename * Update models.py Init Schemas * Update models.py Fix typos. * Update models.py ADD tablenames / fix docstrings * Update models.py Add One-To-Many/One-To-One relationships. * Create __init__.py * Update requirements.txt Add PySyft as a gateway requirement. * Update models.py Finish model schemas * Create database_tests.py Add database unit tests * Update models.py Add syft import * Fix code style * Rename module Rename module : persistence -> storage * Update __init__.py replace import: persistence -> storage * Update routes.py replace import persistence -> storage * Fix code style * Rename database CI tests * Create warehouse.py Add generic data warehouse class. * Update controller.py Rename dict (workers -> cache) / Add storage attribute * Update requirements.txt Fix PySyft reference * ADD host_federated_training event ADD host_federated_training WS event * Update controller.py Add an worker database interface * Update controller.py Add worker database interface * Update models.py Fix docstrings/ add new field at protocol/plan * Update warehouse.py Fix multi args operators * Update fl_events.py Update events * Update controller.py Fix datetime * Update models.py Add ModelCheckPoint.alias * Update controller.py Fix argument operators * Move database_tests ADD database tests to CI unit tests * Update FL Process * Update models/fl_controller * Update controller.py Add PyGrid Exception handling * Create exceptions.py Create PyGrid's Exceptions * Update fl_events.py Improve response messages / remove not necessary print * Update codes.py Add new codes (accepted, rejected, success) * Update routes.py - Add PyGrid Exception Handling - Refactory federated/get-model - Refactory federated/get-protocol - Refactory federated/cycle-request - Remove static messages * Update models.py - Update docstrings - Uncomment syft library * Update controller.py - Add PyGrid's Exception Handling - Add useful get methods * Add syft library * Update fl_events.py - Add unhexlify - deserialize model * Update controller.py - Fix create_process parameter - Remove hard coded cycle's version * Update test_gateway_sockets.py Add test_fl_process * Update test_gateway_sockets - Fix fl_process CI test Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Update gateway Dockerfile (#499) - Fix zstd issue during building docker image Co-authored-by: José Benardi de Souza Nunes <benardinunes@gmail.com> * Remove json.dumps from host_traning_model request (#501) * Remove scope (#505) * Remove Scope module - Legacy Code * Update worker.py Fix typos * Delete webrtc_events.py Remove scope events * Update __init__.py Remove webrtc_events import * Add federated/get-plan route (#482) * Add federated/get-plan route 'receive_operations_as' parameter being ignored * Fix var plan naming * Fix code formatting Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fix request cycle (#503) * Update models.py - Change model id type to auto-increment integer - Rename checkpoints back reference * Update controller.py - Remove pre-defined id - Save initial model checkpoint * Update fl_events.py - Remove model deserialization * Update test_database.py Update Model tests * Fix FL process references - Now, We can retrieve a fl process by using the name defined at client_config * Update test_gateway_sockets.py -Fix FL Process test * Update routes.py - Fix little bug (inverted parameter order) * Update exceptions.py - Add new exception (FLProcessConflict) * Update controller.py - Check if FLProcess / Version already exists - Fix request_cycle bugs * Update models - Add name/version fields at FLProcess * Update warehouse.py - Fix .contains method * Run black tool * Update models.py Add version as a parameter in the FL Process model * Update controller.py - Uncomment model_checkpoints registration - Add version as a parameter during FL Process queries * Update FL unit test - Add version during cycle-request * Update controller.py - Add get_plan method * Update Search method (#508) - Remove custom search method. - Remove syft.search overwrite * PR ready: average a plan (#498) * averages a plan based on diffs Co-authored-by: maddie_shang <mshang@tableau.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Add build context in docker-compose.yml (#510) Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * skip fl tests (#512) * skip fl tests * code formatting * FL Process API fixes (#509) * Fix FL host request; make get-model and get-plan to return binary * formatting with black Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * remove scope tests (#511) * Fix FL tests (#513) * Fix FL tests * Black * Update fl_events.py (#515) * Update fl_events.py Add field "type" on WS responses * Update test_gateway_sockets.py Update tests * Update test_gateway_sockets.py Fix fl tests * add dummy auth endpoint to unblock (#516) * add dummy auth endpoint to unblock * add dummy auth endpoint to unblock Co-authored-by: maddie_shang <mshang@tableau.com> * Update FL report, make diff avg async (#518) * Update FL report, make diff avg async * Fix formatting * Fix unit tests * Fix formatting * Update HTTP responses (#520) * Update routes.py Fix HTTP requests issues. * Update routes.py Code style fix * Update request cycle (#521) Closes #514 * Update controller.py - Add model name and version to rejected responses. - Remove timeout field if it's the last cycle. * Update test_gateway_sockets.py - Add check on version field during request-cycle test * Update requirements.txt (#522) Updating flask and requests to match with syft 0.2.4 * Fix bug in docker-compose.yml file. (#524) When DATABASE_URL changed to REDISCLOUD_URL three slashes were mistakenly kept. * Create gunicorn dev conf (#523) * create gunicorn dev conf * code formatting with black Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Create docs with running instructions for grid gateway and node (#525) * Update README.md * Create README.md * Add running instructions * Update README.md * Add docker section * Update README.md * Update README.md * Updating requirements.txt package versions (#529) * Updating requirements.txt package versions Updating the gateway requirements to match the versions required by PySyft. * removing flaskSocket from requirements.txt FlaskSocket IO is no longer required * auth working (JWT + HSA + RSA) (#517) * auth working * fix placeholder imports * black clashes with pre-commit hook * wip: debug no module named jwt * fix req * RSA and HSA in * RSA and HSA in * add reqs * remove prints * fix db stub * remove prints * clean up * clean up Co-authored-by: maddie_shang <mshang@tableau.com> * Update PyGrid tutorials (#528) * remove heroku tutorials * update intro to pygrid * update intro tutorials * rename tutorials * delete android examples * fix black issues * DELETE outdated tutorials Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fix Docker network interface (#533) * Fix Docker network interface * Update README.md * Move Grid Node to a different repository (#531) * MOVE Grid Node Move Grid Node source code to another repository : https://github.com/OpenMined/GridNode * Update CI tests Comment Grid Nodes tests * FIX code style * Fix code style * Delete unused tests * Remove unused conftest settings * Move PyGrid app source files * Update requirements.txt - Add CI dependencies at requirements.txt file * Update requirements.txt * [REFACTORY] RENAME Rename app gateway -> grid * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update docker-build.yml (#534) Update the "workdir" reference * Connection speed test (#526) * Update routes.py Add connection_speed_test * Update fl_events.py Fix Max Cycle Limit * Update routes.py Fix connection_speed_route * Update routes.py fix typo * Update exceptions.py Add MaxCycleLimiteExceeded Exception * Update routes.py Update test_download / test_upload methods * Update requirements.txt Add requests-toolbelt dependency * Remove unused folder/files * Add requests_toolbelt dependency * Update routes.py - Increase the data sample size to 64 MBytes Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * [REFACTORING] Improving PyGrid's code quality (#535) * Rename storage -> models - Rename module (storage -> models) - Split daatabase models into different source files - Move db singleton instance to models module * Update models module * Update CI test_database * Fix host_federated_training test * Update import references * REFACTORY fl_controller - Move complete_cycle, average_plan_diffs to cycle_manager - Fix worker.is_eligible method - Update report event * Update models module * Update Grid Architecture - Split database models into different modules * Fix small bugs * Fix flake issues * Update imports * Create routes module Split route source file into different source files. - federated - network - general * Update federated.py * Rename module - node_routing -> network * Update routes.federated.py - Replace mockup vars to database commands described by @Prtfw * Trim Plan to torchscript or "list of ops" variants and serve them (#536) * Convert Plan to torchscript and serve it * Host request error handling * Update to follow PySyft changes * Fix cycle manager, make protocol optional in cycle request * Update requirements.txt Co-authored-by: Patrick Cason <me@patrickcason.com> * Update docker-compose.yml (#540) Make it easier to build new gateway image * pre-commit in reqs_dev * fix merge errors * requirements.txt deleted * requirements moved to pip-dep * delete app folder because moved to grid folder * python setup.py install * grid folder * test from grid package * setup files * minor fix * run test using coverage * foo commit * Update to latest PySyft/State and fix typo #555 * Dockerfile and sh updated * the funcionality of grid.py has been moved to __main__.py * version * syft version fixed * description modified * grid.py is in this __main__ * Fix formatting Co-authored-by: iamtrask <liamtrask@gmail.com> Co-authored-by: Bobby Wagner <raw141@case.edu> Co-authored-by: Marianne Linhares Monteiro <mariannelinharesm@gmail.com> Co-authored-by: Jon Lebensold <jon@lebensold.ca> Co-authored-by: Jose A. Corbacho <mccorby@gmail.com> Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> Co-authored-by: Ionésio Junior <ionesio@lsd.ufcg.edu.br> Co-authored-by: Amit Rastogi <amitrastogi2206@gmail.com> Co-authored-by: Vova Manannikov <12518480+vvmnnnkv@users.noreply.github.com> Co-authored-by: Prince Kumar Maurya <prince06.1996@rediffmail.com> Co-authored-by: Tallal Javed <Tallal.javed@outlook.com> Co-authored-by: midokura-silvia <43242089+midokura-silvia@users.noreply.github.com> Co-authored-by: Daniele Gadler <daniele.gadler@yahoo.it> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> Co-authored-by: Héricles Emanuel <hericles@lsd.ufcg.edu.br> Co-authored-by: João Lucas <joao.felipe@ccc.ufcg.edu.br> Co-authored-by: José Benardi de Souza Nunes <benardinunes@gmail.com> Co-authored-by: filippo balicchia <fbalicchia@gmail.com> Co-authored-by: Arjit Chauhan <arjitc12@gmail.com> Co-authored-by: Marcus Vinicius <marcus.costa@ccc.ufcg.edu.br> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: codebecker <chrisbeckerbw@gmail.com> Co-authored-by: George-Cristian Muraru <george.muraru@stud.acs.upb.ro> Co-authored-by: George-Cristian Muraru <murarugeorgec@gmail.com> Co-authored-by: Matthias Lau <lau.matthias@gmail.com> Co-authored-by: systemshift <42102034+systemshift@users.noreply.github.com> Co-authored-by: Igor <sheleztt@gmail.com> Co-authored-by: Ramjee <gantir@users.noreply.github.com> Co-authored-by: Ayoub Benaissa <ayouben9@gmail.com> Co-authored-by: chilang <chilang@gmail.com> Co-authored-by: Madeleine Shang <shang.madeleine@gmail.com> Co-authored-by: Javan Lacerda <javan.lacerda@ccc.ufcg.edu.br> Co-authored-by: Jeferson Ferreira <jefersonnpn@gmail.com> Co-authored-by: maddie_shang <mshang@tableau.com> Co-authored-by: Will <will.badr@gmail.com> Co-authored-by: kasztanka <m.lipowska@onet.pl> Co-authored-by: Patrick Cason <me@patrickcason.com>
Benardi
pushed a commit
that referenced
this pull request
May 12, 2020
* remove heroku tutorials * update intro to pygrid * update intro tutorials * rename tutorials * delete android examples * fix black issues * DELETE outdated tutorials Co-authored-by: Ionésio Junior <ionesiojr@gmail.com>
Benardi
added a commit
that referenced
this pull request
May 12, 2020
* pip-dep folder & requirements_dev.txt * reqs moved to pip-dep folder * Makefiled updated to pip-dep folder * reqs moved to pip-dep folder * git action updated to pip-dep folder * Update run-tests.yml * initial toy websocket based example * removed pycharm files * added support files * Added example client/worker * renamed classes * intermediate commit * file running * initial grid support * added worker discovery to flask server * changed session for sqlalchemy datbase * added known workers to GridClient * checkpoint * initialized grid client and app * clean dev * reinit * a few edits * Added destroy() * added notebook * Added warning * updated warning * updated warning * updated warning * updated warning * Link PySyft in README file (#193) * small bugfix * Fixing heroku check (#194) * fixing heroku check * fixing version check for heroku * syft update * trask_grid -> dev * Fix #196 (#198) * Clean up (#201) * Clean up app.py * Create utils and clean up init * Clean up client * Clean up: part 2 (#212) * Clean up app.py * Create utils and clean up init * Clean up client * Fix typo * Create dev_user and force params * Add verify identity * Fix local redis url * cd to grid folder before checkout dev * Fix verify identity * Adding tests and how to execute them * Rename: tests -> test * Remove force (not working as expected) * Delete test_local_grid_client.py * Moving SocketIO Client and Server Worker from PySyft to Grid (#218) * fixing broken test * moving socketIO workers to grid project * Move socketio (#219) * fixing broken test * moving socketIO workers to grid project * fixing tests * Pre-commit, tests, contributors guideline (#220) * Pre-commit, tests, contributors guideline * PySyft -> Grid * Add Makefile * Examples to run with Android app (#221) * Examples to run with Android app * Fix imports * Add grid websocket approach (#217) * ADD async websocket grid app * ADD sync websocket client class * Update requirements/ change app * Rename App * Change /identity response * Deployment of websocket application * Unify address parameters at WebsocketGridClient * Little change at Procfile to deploy app with flask socket worker config * Update websocket app requirements.txt * Add -k flag at Procfile * Add eventlet worker to manage network app concurrency * [Refactor] Rename flask dir to rest_api * Add WebsocketGridClient class at grid module * ADD binascii operations * Change websocket application * Update requirements.txt * Add jupyter notebook example * Remove var type at docstrings * Create LICENSE * Add python-socketio at requirements list (#231) * Run Black (#233) * Run Black * Add coverage to requirements * Add syft to requirements * Add binder, build status, slack and FOSSA tags (#234) * Update requirements.txt (#237) * Tensor persist (#235) * fixing bug where localhost redis path not good * wip for getting postgres with a schema * moving socketIO workers to grid project * started getting flask integration test working and postgres migration * adding migration * fixing broken test * moving socketIO workers to grid project * fixing tests * got worker -> worker_objects relation working * removing whitespace * removing dead code * format fix * adding flask_sqlalchemy * adding flask_migrate * removing redundant import * unnecessary for now * commented for now * cleaning up imports * removing dead code * cleaning up more imports * added a description * Fix #238 (#242) * Grid communication (#245) * Add communication between grid nodes * Add grid at requirements.txt * Set is_client_worker flag to false / remove auto verbose mode * Run black format tool * Fix websocket app grid dependency at requirements.txt * Change log/debug message at connect grid node callback * Remove know_nodes dict * Performance Optimization between Grid nodes communication (#249) * Performance Optmization * Add notebook to measure performance before/after remove sleep procedure * Remove duplicated notebook * Add reference to Pysyft tutorial * added check for windows for tmp folder deletion (#250) * added check for windows so that tmp folder is deleted only for non-windows platform in case grid already exists * added check for verbose to output command execution status * Add ConnectionError Exception (#252) * Add ConnectionError Exception at WebsocketGrid Client * Fix code style pattern * removed flake8 docstrings from pytest in travis so travis could run (#257) * Add architecture diagram (#254) * Add architecture diagram * Improvements to the diagram * Fix notebook after changes in serde (#255) * rename tutorial file for windows compatibility (#261) Can't checkout file on windows because of ":" in the filename. Also, to be consistent with pysyft tutorials naming style: https://github.com/OpenMined/PySyft/tree/dev/examples/tutorials * REFACTORY Grid Node Application (#264) * Heroku script for creating app instance (#263) * Tensor persist router (#248) * fixing bug where localhost redis path not good * wip for getting postgres with a schema * moving socketIO workers to grid project * started getting flask integration test working and postgres migration * adding migration * fixing broken test * moving socketIO workers to grid project * fixing tests * got worker -> worker_objects relation working * removing whitespace * removing dead code * format fix * adding flask_sqlalchemy * adding flask_migrate * removing redundant import * unnecessary for now * commented for now * cleaning up imports * removing dead code * cleaning up more imports * added a description * starting flask api tests * adding persistence support for grid with unit tests * adding notes * removing unnecessary imports * removing hidden file * ignoring .DS_Store * formatting * renaming test * renaming test * better assertions * adding verbose mode * adding a workbook * green tests * reformatting * changing heroku script to use postgres * adding Flask-Migrate dependency * adding requirements * moving postgres rest api into the app folder * moving create_app * heroku app * reqs * adjusting module imports so that tests and heroku script will both work * fix import * back again * adding migration command * formatting * fix test coverage * formatting * Grid gateway (#267) * Create a initial version of grid gateway component * ADD remotely search method * ADD FederatedClient inheritance at GridNodeClient * ADD node/gateway dockerfiles + docker-compose config * ADD Search / Join Gateway routes * ADD Grid Network class * ADD /search route at grid node * ADD sy.Message() at search method * FIX Code Style * FIX route conflict * REMOVE gateway unused dependencies * FIX bug at GridNetwork search method * UPDATE README.md * UPDATE README.md * ADD new notebooks * FIX merge review requests * Change to Grid@dev at requirements.txt * run black tool * Update websocket_client.py * Update websocket_client.py * FIX REST_API_TEST (Update to sy.Message structure) * FIX connect_grid_node BUG / Run black tool * ADD WebsocketClient/Grid tests * ADD flask_cors package at requirements file * ADD eventlet package at requirements * ADD docker documentation reference at README.md file * FIX time interval at websocket/grid tests * UPDATE README.md / Decrease time interval between connections * FIX GridNetwork connection bug * REFACTORY tests * run black tool * REMOVE SocketIOAPITest class (change all test cases to pytest design) * REMOVE time.sleep() from socketio test module * run black tool * REMOVE stress test * Revert "REMOVE stress test" This reverts commit 5e330fb051dd7f4845d0f5ab5ca4baf25f3cbb3b. * Remove parameters from connect_nodes test * FIX test_connect_nodes * Update requirement files - Fix a stable version to syft lib - Remove redundant dependencies - Add requests lib at gateway requirements * Change Dockerfile's base image - python-slim base image is more lighter than ubuntu image * ADD new environment variable at gateway compose configs - Now, the gateway component will receive the secret key by environment variable * Revert changes at requirements.txt (websocket_app) - Download syft/torch packages directly from pip spend much disk space. - We have limited space on heroku platform (500 MB) * [FIX] Problem to finish grid node processes after finish test cases * run black tool * ADD time interval between websocket tests - Limited hardware resources to test environment * ADD time interval between node connections * Review requests * Deploy module (#273) * ADD deploy module - ADD BaseDeployment class (abstrac class used to initialize generic attributes) - ADD NodeDeployment class (class used to perform grid node applications deployment) * Update "Launch Websocket Grid Node" notebook - Now, perform grid node app deployment with deploy module * FIX REST_API_TEST (Update to sy.Message structure) * Sync with syft repository - In the last syft commit, the AbstractTensor class was moved to generic module. * Update requirements.txt list - Install the last version of syft library. * UPDATE requirements.txt - Add egg-name to syft package * Revert "Sync with syft repository" - This reverts commit d4c8b4f777b06438e6740392cc901f6b75fb72f0. - These changes have not yet been applied in the syft release version. * Revert changes on requirements.txt * Review requests * RENAME o -> output * FIX undefined variable name * FIX wrong var name * FIX undefined var * RENAME component -> base_deployment source file * Check if gateway_url is not None before using it + fix tests (#276) * Check if gateway_url is not None before using it * import AbstractTensor from syft.generic.tensor * Update package imports (#275) * Updated README.md (#278) * Update README.md With dev branch tutorials and projects page. * Update README.md * Launch on Heroku Update (#280) * __init__.py: - Fixed a typo in execute_command(command) - Added documentation to launch_on_heroku function - Removed unessacery if from launch_on_heroku utils.py: - Added documentation * Updated Part1 example notebook * Update __init__.py * Update utils.py * ran black for formatting on grid * Grid Node Persistence (#277) * ADD SQLAlchemy to grid websocket application - Adapt Jon's persistence features to websocket context. * Check if gateway_url is not None before using it + fix tests (#276) * Check if gateway_url is not None before using it * import AbstractTensor from syft.generic.tensor * Update package imports (#275) * Implement persistence with SQL Alchemy - Refactory SQLAlchemy Models - Remove set ID websocket event - Add recover_worker / snapshot methods * Revert set-grid-id endpoint - To the future, we'll remove this endpoint (This is not the purpose of this pull request). - Each grid node must initialize with its own id, an external component cannot change it. * MOVE db module to main package * run black tool * Snapshot method performance optmization - Before, for every command, we deleted ALL items registered on database and added ALL items again. - Now, for every command, we add/remove the difference between current and last snapshot. * RENAME db_module -> persistence * ADD doc comments * Updated README.md (#278) * Update README.md With dev branch tutorials and projects page. * Update README.md * FIX review requests * Refactor tests (#283) * IDs are lowercase and use a less common gateway port * Create folders * Clean up grid search tests * Update docker environment (#286) * Fix #271 and #287 (#285) * Fix #271 * Increase sleep time * Undo changes in test_api_socketio * Rename test * decrease sleep time * Add test for connect_all_node and fix #287 * Host model on grid node (#272) * Create port argument for websocket_app * Websocket Client inherits from Client * Create serve model route * Create experimental notebook * Return proper response * Create route for model prediction * Create _send_get * Create run_inference function * Rename model_name -> model_id and list models * Move model related methods to client * Update notebook * Clean up warnings * Create unhexlify argument * Increase ping timeout and ping interval * Support plans and jit models * Delete 2. Host a model on a grid node.ipynb * Delete Host a model on a grid node.ipynb * Organize examples folder * Improve tutorials * Move tutorials from experimental * Clean up encoding * Re-run notebook * Add tests * Update client.py * ADD support to models on Grid Gateway (#291) * Increase Grid Network support to models - ADD /choose-model-host route at gateway - ADD /search-model route at gateway - ADD /search-available-models route at gateway - ADD /search-available-tags route at gateway - ADD /dataset-tags route at websocket app - Create a new notebook * ADD unit tests (host jit model / query_model) * Improve gateway's index page * Add random.sample at choose hosts endpoint * Return the first match_node * Revert "Improve gateway's index page" This reverts commit 83af10108bcdb8e50d9a4684d9a04d87faaad734. * Solve review requests - Change list objects to set objects - Fix typo - Set ConnectionError Exception explicitly - Move n_replica's var to gateway.py * ADD host/query plans test * run black tool * Solve review requests - Remove if at gateway.py - ADD unit tests to /search-available-tags, /search-available-models endpoints * ADD run_inference method at grid - ADD run_inference method - Update Notebook - Update tests * Change host_model -> serve_model * Update Notebook * UPDATE grid tests * HOST GPT-2 (#292) * ADD new Notebook * Fix Notebook * Fix typo * ADD requests-toolbelt dependencies * ADD Text generation at Notebook * Solve review requests * Update requirements.txt (#294) - Fixes deployment issues on Heroku with size - Added Postgres driver * Host and Query an Encrypted model (#288) * Create 3. Host and Query an Encrypted model * Rename 3. Host and Query an Encrypted model to 3. Host and Query an Encrypted model.ipynb * Change model to convnet * Use a plan * Support fetching a plan * Remove unused notebook cells * Restart notebook to show that fetch is working properly * Use convnet * Get pointers instead of getting objects * Update for changes in fetch * Add id to model * Support private model hosting and querying * Create a new notebook * Fix notebook * Update reference to correct attribute * Clean up and add comments * Unify serve model and document get_private_model * Add persistent models (#282) * Create port argument for websocket_app * Websocket Client inherits from Client * Create serve model route * Create experimental notebook * Return proper response * Create route for model prediction * Create _send_get * Create run_inference function * Rename model_name -> model_id and list models * Move model related methods to client * Update notebook * Clean up warnings * Create unhexlify argument * Increase ping timeout and ping interval * Persistant Model Storage - model_manager.py for saving, deleting, listing and in short handling all the models - routes.py: updated for model manager and created a new route of deleting a model - jupyter notebook example updated with the delete function and some commenting to make it easy to understand - client.py updated with the delete_model route * A sample pickle model save file. * Support plans and jit models * Delete 2. Host a model on a grid node.ipynb * Delete Host a model on a grid node.ipynb * Organize examples folder * - Added database * Improve tutorials * Move tutorials from experimental * Clean up encoding * Re-run notebook * Add tests * deleted model files. As they are shifted to db. * Improved Comments - Improved comments - Removed conflicts - Improved exception handling * model cache - removed model list from routes - Adding model cache to model_manager. * removed todo * ran black. * Style Fixes - Style fixes - model_manager now returns dicts - Updated client to handle new requests - Success status added in model manager and flask responses. * Merged with dev - Merged with dev - Ran black - Removed unnecessary prints * fixed bug in check_if_model_exists - fixed bug in check_if_model_exists. - better comments and return dicts. * Use LargeBinary instead of BYTEA * Rename MLModel -> TorchModel * Style fixes and fix clear cache * Add missing json loads * Add delete operation * Remove experimental notebook * Add missing json.loads * Remove unused variable and add comment * Add TorchTensor * Create local_worker var * Create file with local worker utilities * Cache real object + fix plans db support * Use local worker utilities * Add tests for deleting a model * Use encoding * Fix GPT-2 demo * Rename fetch_plan to fetch_plan_reference * Fix imports * Use sy.messaging * Fix Message * Fix imports * Create notebook 2.2 * Delete 3. Host and Query an Encrypted model.ipynb * Avoid error during snapshot * Host and Query encrypted models options * Remove fetch_plan_reference and demo 3.3 * Remove get_private_model and demo 3.1 * Rename notebook * Fix make_helper jit support * Add db support for encrypted models * Use local_worker utilities * Clean up model manager * Relaocate comments * Fix issues in Syft for now * Fix torchscript db issue * Add test test_host_encrypted_model * Fix test_host_encrypted_model * Replace 2.2 notebook * Add test_run_encrypted_model * Clean up websocket app and gateway app starting scripts (#296) * Fix typo (tst_config -> test_config) * Use parser and os environ vars as default * Fix import order and typos * Define warning * Fix tests * Update requirements.txt (#297) - fetching syft from dev branch directly. * Clean up host model API (#298) * Add new columns to TorchModel * Add allow_model_copy and allow_run_inference args * Snapshot shouldn't include model objects * Fix tests * Create get_serialized_model_with_id * Create get_model route * Create get_model_copy * Create global variables for storing error messages * Rename allow_model_copy -> allow_get_model_copy * Fix typo * Implement serve_encrypted_model * Improve comments * Add tests for new arguments * Fix test * Standardize results * Update tests * Fix small bug * Improve delete error messages * Fix typo * Fix test * Extend syft with a friendly api * Better argument names * Try to fix tests * Try to fix tests * Fix tests * Added TODO * Update notebooks * download -> download_model * Wrap host/inference encrypted models (#300) * Wrap host/inference encrypted models - Wrap encrypted serve_model / run_inference - ADD /search-encrypted-models at websocket app - ADD /search-encrypted-model / /choose-encrypted-host at grid gateway * Fix typo * ADD encrypted serve_model/run_inference test * FIX serve_model/run_inference test * Fix typo / ADD constant value * Fix review requests * Update requirements.txt (#297) - fetching syft from dev branch directly. * Run black tool * ADD comment about eventlet money_patch * ADD new notebook * Fix heroku ws app (#302) * Improve 3.2 * Fix heroku app * Get rid of main * Update notebook * Fix websocket_app.py * args.gateway_url -> gateway_url * Change Model ids (db.Integer -> db.BigInteger) * Update notebook * Remove outdated comment * Add Introduction Notebooks (#303) * Start What's Grid Notebook * Update and rename 1 - What's Grid?.ipynb to Part 0 - Introduction to Grid * Create Part 1 - Launch a Grid Node Locally.ipynb * Update and rename Part 1 - Launch a Grid Node on Heroku.ipynb to Part 2 - Launch a Grid Node on Heroku.ipynb * Update Notebook (Launch a Grid Node on Heroku) * ADD .ipynb extension at notebook * FIX Launch a Grid Node on Heroku notebook * ADD deployment support to grid gateway * Fix Procfile / gateway.py * Fix websocket_app bug * ADD heroku_gateway deployment module * Improve notebooks * Update demo (#305) * Improve 3.2 * Update demos * Update notebooks * Add note about checking tutorials * Remove Serving models * Review examples (#307) * improve readability of tutorials (#308) * Fix tutorial serving querying (#310) * Fix for psycopg2 package installation on Linux * Fixed torch 1.0.1 requirement for Linux and Windows * Fixed imports' names for the gp2_utils file and added explanation on how to correctly parse the HMNIST dataset * Reverted changes related to dependencies for torch * Reverted psycopg fix * small improvements in readme and tutorial (#313) * small improvements in readme and tutorial * correcting typos and adding some explanations * Fix typos in tutorials (#309) * Fix typos * Update Part 1 - Launch a Grid Node Locally.ipynb * Fix typo * Fix test generation * General fixes / improvements * ADD Gateway persistence (#314) * ADD Gateway persistence * ADD Persistence at Gateway Deployment * ADD Notebook * Update 4. Introducing the Grid Network.ipynb (#316) * Web Index page update (#317) * web initial * model list being fetched from server * finalized web index page * ran black * changed gradient background - to cover. * added comment for model.pop explanation. * fixed typo - removed test endpoint. * ran black * ADD webpage section (#322) * Fix for psycopg2 and torch 1.0.1 package installation on Linux (#301) * Fix for psycopg2 package installation on Linux * Fixed torch 1.0.1 requirement for Linux and Windows * Fixed PyTorch package version for Python 3.7 * Added python runtime environment for Heroku * Disallow cross-origin resource sharing to webpage routes (#323) * Add README files for app folders (#324) * Improve 3.2 * Add README.md files in app * Add grid logo to Readme (#329) * Update README.md added logo and changed grid to PyGrid. * Update README.md * Update model_manager.py (#331) fixes #318 - in case where save_model gives an integrity error * Common Install Issues Section (#334) * Common Installatoin Issues Section fixes #306 * Update README.md * Add api documentation (#328) * Verify download permissions at /get_model/<model_id> route - If someone performs a request to this endpoint outside of grid lib context, they probably will get the model (Security Problem). - Now, /get_model/<model_id> endpoint returns 403 (Forbidden) if not allowed to download. - Now, /get_model/<model_id> endpoint returns 404 (Not Found) If not found at the grid node. * CHANGE /model/<model_id> request method GET -> POST We can forward HTTP body content at GET requests, but it's a bad practice. We should use POST/PUT methods if we want to forward important data between REST API. * Change response error code at /serve-model - Replace 500 Internal Server Error -> 409 Conflict 5xx rest codes are used to represent server errors. 4xx rest codes are used to represent client errors. In this case, our response error message will be sended when we have a already registered model This error was be created by the client side, not server. * ADD Grid Node REST API documentation * Fix markdown * Fix grid_node_rest_api document * ADD Grid Node Websocket API Documentation * ADD gateway REST API * FIX gateway persistence bug * FIX Notebooks examples * FIX api documents * FIX gateway_api document * Merge grid clients (#333) * Merge WebsocketGridClient and GridClient - Move all GridClient's methods to WebsocketGridClient * Remove GridClient class - Remove GridClient reference. - Remove remove GridClient tests. * FIX flake error * Merge Grid Node Apps (#335) * Remove unused apps * Remove outdated/unused tests * Remove notebook(copy) * Socket Grid Node (#338) * [REFACTORY] Grid Node App - Change SocketIO protocol to Pure Websockets Protocol (Now we can send/receive messages from syft.js) - Move some HTTP routes to socket API - Allow to receive/send binary contents (Perfomance Improvement) - Add event_routes source file (handle socket requests) * [REFACTORY] Grid Lib - Remove unused/outdated source file (grid/cliemt.py) - Extends syft.WebsocketClientWorker - Change HTTP node requests to socket node requests - Delete unused/outdated methods * [CHANGES] Tests - Change conftest (update to test new grid_nodes) - Rename test_api_socketio -> test_api_socket - test_api_sockets changes (update to test new grid_nodes) - test_api_grid changes (update to test new grid lib) * ADD gevent/gevent-websocket at requirements.txt * Updaate Procfile * Update requirements dependencies * Fix for psycopg2 on gateway (#344) Replaced the `psycopg2` package with `psycopg2-binary` in the gateway/requirements.txt file. * Update travis badge (#346) * Update travis badge Replaced `https://travis-ci.org/OpenMined/Grid` to `https://travis-ci.org/OpenMined/PyGrid` * Skip test_grid_search Add @pytest.mark.skip decorator at test_grid_search unit test * fixing formatting error * ordered requirements to install grid (#347) * Update the PyGrid components web user interface (#348) * update titles from 'Grid' to 'PyGrid' in web interface * create web interface for pygrid network based on node interface * Add Issue Templates (#349) * create issue template: feature request * create issue template: bug report * Add sphinx config to build docs (#351) * Add sphinx config to build docs Edit conf.py to set rtcat theme. Set napoleon to allow Google style pydocs. * Add documentation build to travis CI verifications (#353) * add new parameter to docker-compose (#340) * add dataset tags table and fix ui grid (#357) * Update index.html (#358) * Update template.css (#361) * Add setup.cfg/pbr and make package Ready-to-Publish (#362) * Add pbr and setup.cfg * Edit setup.cfg and README for valid pypi package Edit invalid setup fields. Set README image to hosted files at Github. * Add Pull Request template (#364) * Add Pull Request template * Edit PR template to clarify requests * Edit PR template to remove typo * Update PyGrid-Arch Image (#365) * Add PyGrid-Arch image * Update PyGrid Architecture Image on README * Authentication (#342) * ADD Auth module at grid lib * Update grid __init__.py * ADD Auth module at Grid Node App - Create Session Repository - Create UserSession - Socket Session Auth wrapper - Authenticate socket route * ADD authenticate method * SKIP test_host_plans * Fix problem with _load_preset_credentials * ADD flask_login lib * Handle Multiple Private/Public Workers - Now we're able to connect using different private workers at the same node. - Map private workers by user attestation. - We're able to connect with local private/public workers. - We're able connect private/public workers with remote nodes. * Remove unused source file * ADD docstrings * Remove custom search method at grid node * Remove search method at grid.websocket_client * Remove unused imports * Update Authenticate event response * Move model_manager source file to persistence module * Fix /workers/ endpoint * ADD default admin infos * Update auth module * Fix search method * RENAME create_standard_auth -> register_new_credential * RENAME load_auth_credentials -> load_credentials * ADD docstrings at auth module * MOVE 'set_database_config' to persistence module * REMOVE local_worker_utils source file * ADD docstrings at PyGrid App * REMOVE outdated functions 'launch_on_heroku' * FIX/ADD Docstrings at PyGrid lib * CREATE grid_codes source files * REFACTORY events - Create events module at PyGrid Node App - events.py -> events/__init__.py - event_routes -> syft_events.py, model_events.py, control_events.py * Update grid_codes.py * Skip Plans CI tests - PySyft last update crash Plans CI tests/ grid functions, it will fixed in another PR. * Fix flake/black code style errors * Add entrypoint.sh (#368) * route to delete a grid node (#367) * route to delete a grid node * fixing routes with black * runing black at manager file * Bump werkzeug from 0.15.2 to 0.15.3 in /gateway (#369) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.15.2 to 0.15.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/0.15.2...0.15.3) Signed-off-by: dependabot[bot] <support@github.com> * Adding MNIST Federated Example (#366) * Adding MNIST Federated Example * Cleaning notebooks outputs/Adding cuda support to mnist federated example * Adding empty folder to download datasets / Changing datasets path * Updating files permissions * Bump werkzeug from 0.15.2 to 0.15.3 in /app/websocket (#370) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.15.2 to 0.15.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/0.15.2...0.15.3) Signed-off-by: dependabot[bot] <support@github.com> * Edit docs to serve it remotely (#375) * Mock deps for autodoc * Replace autodoc for autoapi Set autoapi to averts heavy dependencies * Edit sphinx conf to enforce theme css * Edit sphinx to include generated api documentation * Conform code to black standards * Fix file permissions (#376) * updating websocket route event (#378) * Update local examples to conform to grid module (#382) * Fix serde and logging imports in Android worker example notebook (#380) * Move documentation/ files into docs hierarchy (#384) * Updated torch version for worker container (#386) * Replace Grid with PyGrid (#387) * Add spam Example (#388) * Add files from SPAM Folder * Spam Populate Grid Network Notebook * Spam Train Model * Black reformat for Spam example * Spam fix typo (#391) * Fix bug on remote inference (#392) * updating websocket route event * undo commit * fixin response message on websocket app * Start Android example server on `0.0.0.0` to accept all connections (#394) Change address to '0.0.0.0' to accept all connections * Download model bug (#396) * updating websocket route event * undo commit * changing way to get serialized model * Change PySyft branch from dev -> master (#397) PySyft no longer hosts its latest code in the dev branch. We push straight to master. * Remove unused/outdated import (#398) * Refactory gateway Routes (#403) * Gateway Routes Refactory - Decrease Cyclomatic complexity. - Avoid/Handle Internal Server Errors * ADD INVALID_JSON_FORMAT_MESSAGE constant * Fix the websocket app procfile module to websocket_app (#404) * Rename folders used in heroku deploy scripts from grid to PyGrid (#406) * Add missing proto dependency to the websocket app (#405) * Process tensor request (#399) * Remove unused/outdated import * Handle GetNotPermittedError and save tensor request * Add CI tests * Update PySyft repository reference * Fix requirements.txt * Fix .travis install session * Add egg information to requirements.txt (#409) On my machine (OSX 10.14.6 running python 3.6), while I was able to install requirements.txt using 'pip install -r requirements.txt', I was not ablee to do so by running 'pip install .' as the README directs. After some googling, adding egg information to the github imports in requirements.txt allowed me to do so. I don't claim to be an expert on this issue, but it worked for me. * remove_Connecting_Locally.ipynb (#414) * Decode error (#400) * updating websocket route event * undo commit * adding try catch to decode bytes that utf-8 cant decode * [FIX] serve_models / run_encrypted_inferences on plans. (#420) * [GridNode] Fix host/run_inference on plans * [PyGrid Lib] Update serve_model method. * [CI Tests] Allow / Update tests using plans. * Fix encrypted inference route * [CI Tests] Allow/Update host/run_inference/run_encrypted_inferences at grid tests. * Fix requirements.txt for heroku The syft_proto is currently failing to install on Heroku with the HerokuNodeDeployment script. Trying to fix it * Use PyPI version of syft_proto * Bugfix * Bugfix * Fix app requirements.txt (#422) * [Bug FIX] Standard tensors also have the get method, even without children. (#425) * Fix requirements hierarchy (#423) * Fix app requirements.txt * Remove proto dependency from requirements.txt, fix dependency hierarchy (it'll avoid package incompatibilities between syft/pygrid and syft-proto) * Update docker-compose.yml (#424) * Update docke-compose / README * Fix README.md * Correct Fell->Feel type in the README. (#426) There is a noticeable typo. Please accept this patch to fix it. * Kubernetes Config file (#415) * docker files * Updated commands formatting * Added services, removed unnecessary entries from deployment * connecting to the local gateway service * moved files to appropriate base folders * rationalised the label names, updated image of node to default * add gateway overlays for dev and prod. * basic kustomize working for gateway * started using config maps for env vars * used configmaps for node * deploy script * updated service to be available externally * Fixed the deploy to proper shape * added todo, renamed k8s to readme * updated readme.ms * Update todo.md * added docker build grid-gateway, grid-node * removed changes for to travis.yml * update docker image * support for multiple nodes * support for multiple nodes in prod * support for multiple nodes in test Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Updated k8s Readme (#427) * docker files * Updated commands formatting * Added services, removed unnecessary entries from deployment * connecting to the local gateway service * moved files to appropriate base folders * rationalised the label names, updated image of node to default * add gateway overlays for dev and prod. * basic kustomize working for gateway * started using config maps for env vars * used configmaps for node * deploy script * updated service to be available externally * Fixed the deploy to proper shape * added todo, renamed k8s to readme * updated readme.ms * Update todo.md * added docker build grid-gateway, grid-node * removed changes for to travis.yml * update docker image * support for multiple nodes * support for multiple nodes in prod * support for multiple nodes in test * Updated the docker build image name * updated Readme.md with kubernetes, docker image name Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Add mpc flag / column for encrypted models (#428) * Replacing Grid Node's PyGrid/PySyft dependencies (#429) * REFACTORY: replace lib (grid -> syft) at grid node app. * REFACTORY: replace lib (grid -> syft) at CI tests. * Update syft temporarily * remove grid imports * ADD private_grid unit tests * Update requirement files * Remove unused imports * Add import warning Co-authored-by: Andrew Trask <liamtrask@gmail.com> * [Refactory] Grid node's persistence module (#432) * Remove SQLAlchemy models / source files * Refactory Persistence module * Fix syft remote search * Update websocketapp / syft_events - Update websocketapp to receive redis url by args / environment var - Update syft_events to import object_storage and use it to recover worker * Fix get model from cache * Remove model instance checkup * Remove save/get plan states * Update model events * Fix/Update persistence module * Update CI tests configs * Update gitignore / add database source file * Run black tool * Update requirements.txt * Update requirements.txt * Change event error message * storage cache encapsulation * Fix/Update routes * Update database server * Replace literal strings by codes * Replace literal strings by syft.codes * Add docstrings * Updating grid status and name at grid app (#443) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * Grid Standardization (#447) * starting gateway implementation over websockets * create ws blueprint * fix some imports * Fix flake/black issues * Add ws events sketch * - Create Scope Class - Create Scope Controller - implement get-protocol message * create codes for response * Add socket handler to manage connections * Fix get-protocol endpoint glitches * Implement webrtc routes * fix black errors * Add docstrings * create test file for gateway websockets api * add socket client for test mocking * update gateway app runner on conftest and fix socket ping endpoint * update requirements.txt with aiounittest * add socket_ping test * add some tests for gateway websockets * fix black issues Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Setup GitHub Actions Workflow (#448) * setup github actions workflow * exclude syft * exclude syft * update syft repo * skip some tests and update syft origin * remove travis and update build status badge (#449) * Consistent Model Hosts (#450) * Search nodes for ones already hosting the model when choosing model host * Add a test for model host consistency * Improve k8s documentation (#452) * Update requirements.txt (#442) Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Create Federated Learning Processes (#458) * create a draft of a federated learning process class * create a controller to manage federated learning processes * fix black issues * update classname and make client_protocols a default arg * Use docker port binding instead of host networking (#461) Solves #460 * working docker compose after removing extra '/' in REDISCLOUD_URL. Both notebooks https://github.com/OpenMined/PySyft/blob/master/examples/tutorials/grid/federated_learning/mnist/ should works now Need to add 127.0.0.1 -> host.docker.internal to /etc/hosts to work around Docker for Mac not supporting network_mode: 'host' (#462) Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Create Workers (#470) * create possible user class * create worker concept * WIP: fl_download_model_endpoint (#471) * Change PySyft Reference (#474) * Add flask ws worker as a Docker ENTRYPOINT (#475) * Remove outdated grid library source code (#476) * Remove outdated grid lib * Remove library documentation * remove unused file * Remove grid lib config files * Update Git CI tests * Remove outdated tests using grid library * Update README.md (#477) * Update README * Add PyGrid image * Remove image from README.md * remove readthedocs file * Updating Launch examples (#478) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * adding route to download protocol * getting the issue done * fixing merge to PyGrid dev * download protocol http endpoint * download protocol http endpoint * download protocol http endpoint * finishing issue #446 * deleting federated route * updating examples how launch grid * changing python version * changing python version * changing python version part3 * changing python version part3 * changing python version part4 Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Sketch Events for Mobile/Web Federated Learning (#469) * Refactory pygrid codes * Remove scope events * Add fl_events sketch * Update events module * Skip scope tests * Fix code style * Remove unused codes * Fix expected JSON fields * Add model_id / remaining_time * Fix code style * Add FederatedLearningCycle class * Fix imports * Minor fixes * Fix Flake8 issues * Fix black tool issues * Support Cycle sequences * Rename FLPController -> FLController * Fix import FLPController -> FLController * Protocol download endpoint (#480) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * adding route to download protocol * getting the issue done * fixing merge to PyGrid dev * download protocol http endpoint * download protocol http endpoint * download protocol http endpoint * finishing issue #446 * deleting federated route * updating examples how launch grid * changing python version * changing python version * changing python version part3 * changing python version part3 * changing python version part4 * adding new route to get protocols * changing branch * fixing black * changing protocols to cliente_protocols Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Setup GH Actions Workflow for Automated Docker build (#484) * Create docker-build.yml * Update run-tests.yml * Delete docker-build.yml * Create docker-build.yml * Create docker-build.yml * Update docker-build.yml * Update docker-build.yml * Update docker-build.yml * remove conditional * update grid node path * Update CI badges in README (#485) * Create ansible playbook to deploy the pygrid (#472) * Create ansible playbook to deploy the pygrid * Refactor playbook/README and add destroyer playbook * update README with destroy playbook usage * remove unecessary vars reference Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Update README and Contributing guidelines (#488) * Update links and numbers * Update CONTRIBUTING.md * Update requirements.txt (#492) * Update Docker Publish Workflow (#487) * Update docker-build.yml * Rename Job Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fl report endpoint (#489) * report endpoint Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * pr for feedback fl_impFLcycles - use server config (#479) * WIP: fl_download_model_endpoint * mock up use of server_config params for #438-2 * mock up use of server_config params for #438-2 * math stuff * wip: model in place * wip: model in place * wip: model in place * wip: model in place * wip: model in place * comment code * comment code * comment code Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Add FL http cycle request endpoint (#497) * [WIP] add federated/cycle-request endpoint to gateway routes * [WIP] enable worker to get cycles and update cycle-request endpoint * no changes. just to avoid overwritten by merge * (fix) remove get_cycles function * PyGrid Gateway Persistence (#481) * Initial commit * Fix tablename * Update models.py Init Schemas * Update models.py Fix typos. * Update models.py ADD tablenames / fix docstrings * Update models.py Add One-To-Many/One-To-One relationships. * Create __init__.py * Update requirements.txt Add PySyft as a gateway requirement. * Update models.py Finish model schemas * Create database_tests.py Add database unit tests * Update models.py Add syft import * Fix code style * Rename module Rename module : persistence -> storage * Update __init__.py replace import: persistence -> storage * Update routes.py replace import persistence -> storage * Fix code style * Rename database CI tests * Create warehouse.py Add generic data warehouse class. * Update controller.py Rename dict (workers -> cache) / Add storage attribute * Update requirements.txt Fix PySyft reference * ADD host_federated_training event ADD host_federated_training WS event * Update controller.py Add an worker database interface * Update controller.py Add worker database interface * Update models.py Fix docstrings/ add new field at protocol/plan * Update warehouse.py Fix multi args operators * Update fl_events.py Update events * Update controller.py Fix datetime * Update models.py Add ModelCheckPoint.alias * Update controller.py Fix argument operators * Move database_tests ADD database tests to CI unit tests * Update FL Process * Update models/fl_controller * Update controller.py Add PyGrid Exception handling * Create exceptions.py Create PyGrid's Exceptions * Update fl_events.py Improve response messages / remove not necessary print * Update codes.py Add new codes (accepted, rejected, success) * Update routes.py - Add PyGrid Exception Handling - Refactory federated/get-model - Refactory federated/get-protocol - Refactory federated/cycle-request - Remove static messages * Update models.py - Update docstrings - Uncomment syft library * Update controller.py - Add PyGrid's Exception Handling - Add useful get methods * Add syft library * Update fl_events.py - Add unhexlify - deserialize model * Update controller.py - Fix create_process parameter - Remove hard coded cycle's version * Update test_gateway_sockets.py Add test_fl_process * Update test_gateway_sockets - Fix fl_process CI test Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Update gateway Dockerfile (#499) - Fix zstd issue during building docker image Co-authored-by: José Benardi de Souza Nunes <benardinunes@gmail.com> * Remove json.dumps from host_traning_model request (#501) * Remove scope (#505) * Remove Scope module - Legacy Code * Update worker.py Fix typos * Delete webrtc_events.py Remove scope events * Update __init__.py Remove webrtc_events import * Add federated/get-plan route (#482) * Add federated/get-plan route 'receive_operations_as' parameter being ignored * Fix var plan naming * Fix code formatting Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fix request cycle (#503) * Update models.py - Change model id type to auto-increment integer - Rename checkpoints back reference * Update controller.py - Remove pre-defined id - Save initial model checkpoint * Update fl_events.py - Remove model deserialization * Update test_database.py Update Model tests * Fix FL process references - Now, We can retrieve a fl process by using the name defined at client_config * Update test_gateway_sockets.py -Fix FL Process test * Update routes.py - Fix little bug (inverted parameter order) * Update exceptions.py - Add new exception (FLProcessConflict) * Update controller.py - Check if FLProcess / Version already exists - Fix request_cycle bugs * Update models - Add name/version fields at FLProcess * Update warehouse.py - Fix .contains method * Run black tool * Update models.py Add version as a parameter in the FL Process model * Update controller.py - Uncomment model_checkpoints registration - Add version as a parameter during FL Process queries * Update FL unit test - Add version during cycle-request * Update controller.py - Add get_plan method * Update Search method (#508) - Remove custom search method. - Remove syft.search overwrite * PR ready: average a plan (#498) * averages a plan based on diffs Co-authored-by: maddie_shang <mshang@tableau.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Add build context in docker-compose.yml (#510) Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * skip fl tests (#512) * skip fl tests * code formatting * FL Process API fixes (#509) * Fix FL host request; make get-model and get-plan to return binary * formatting with black Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * remove scope tests (#511) * Fix FL tests (#513) * Fix FL tests * Black * Update fl_events.py (#515) * Update fl_events.py Add field "type" on WS responses * Update test_gateway_sockets.py Update tests * Update test_gateway_sockets.py Fix fl tests * add dummy auth endpoint to unblock (#516) * add dummy auth endpoint to unblock * add dummy auth endpoint to unblock Co-authored-by: maddie_shang <mshang@tableau.com> * Update FL report, make diff avg async (#518) * Update FL report, make diff avg async * Fix formatting * Fix unit tests * Fix formatting * Update HTTP responses (#520) * Update routes.py Fix HTTP requests issues. * Update routes.py Code style fix * Update request cycle (#521) Closes #514 * Update controller.py - Add model name and version to rejected responses. - Remove timeout field if it's the last cycle. * Update test_gateway_sockets.py - Add check on version field during request-cycle test * Update requirements.txt (#522) Updating flask and requests to match with syft 0.2.4 * Fix bug in docker-compose.yml file. (#524) When DATABASE_URL changed to REDISCLOUD_URL three slashes were mistakenly kept. * Create gunicorn dev conf (#523) * create gunicorn dev conf * code formatting with black Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Create docs with running instructions for grid gateway and node (#525) * Update README.md * Create README.md * Add running instructions * Update README.md * Add docker section * Update README.md * Update README.md * Updating requirements.txt package versions (#529) * Updating requirements.txt package versions Updating the gateway requirements to match the versions required by PySyft. * removing flaskSocket from requirements.txt FlaskSocket IO is no longer required * auth working (JWT + HSA + RSA) (#517) * auth working * fix placeholder imports * black clashes with pre-commit hook * wip: debug no module named jwt * fix req * RSA and HSA in * RSA and HSA in * add reqs * remove prints * fix db stub * remove prints * clean up * clean up Co-authored-by: maddie_shang <mshang@tableau.com> * Update PyGrid tutorials (#528) * remove heroku tutorials * update intro to pygrid * update intro tutorials * rename tutorials * delete android examples * fix black issues * DELETE outdated tutorials Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fix Docker network interface (#533) * Fix Docker network interface * Update README.md * Move Grid Node to a different repository (#531) * MOVE Grid Node Move Grid Node source code to another repository : https://github.com/OpenMined/GridNode * Update CI tests Comment Grid Nodes tests * FIX code style * Fix code style * Delete unused tests * Remove unused conftest settings * Move PyGrid app source files * Update requirements.txt - Add CI dependencies at requirements.txt file * Update requirements.txt * [REFACTORY] RENAME Rename app gateway -> grid * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update docker-build.yml (#534) Update the "workdir" reference * Connection speed test (#526) * Update routes.py Add connection_speed_test * Update fl_events.py Fix Max Cycle Limit * Update routes.py Fix connection_speed_route * Update routes.py fix typo * Update exceptions.py Add MaxCycleLimiteExceeded Exception * Update routes.py Update test_download / test_upload methods * Update requirements.txt Add requests-toolbelt dependency * Remove unused folder/files * Add requests_toolbelt dependency * Update routes.py - Increase the data sample size to 64 MBytes Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * [REFACTORING] Improving PyGrid's code quality (#535) * Rename storage -> models - Rename module (storage -> models) - Split daatabase models into different source files - Move db singleton instance to models module * Update models module * Update CI test_database * Fix host_federated_training test * Update import references * REFACTORY fl_controller - Move complete_cycle, average_plan_diffs to cycle_manager - Fix worker.is_eligible method - Update report event * Update models module * Update Grid Architecture - Split database models into different modules * Fix small bugs * Fix flake issues * Update imports * Create routes module Split route source file into different source files. - federated - network - general * Update federated.py * Rename module - node_routing -> network * Update routes.federated.py - Replace mockup vars to database commands described by @Prtfw * Trim Plan to torchscript or "list of ops" variants and serve them (#536) * Convert Plan to torchscript and serve it * Host request error handling * Update to follow PySyft changes * Fix cycle manager, make protocol optional in cycle request * Update requirements.txt Co-authored-by: Patrick Cason <me@patrickcason.com> * Update docker-compose.yml (#540) Make it easier to build new gateway image * pre-commit in reqs_dev * fix merge errors * requirements.txt deleted * requirements moved to pip-dep * delete app folder because moved to grid folder * python setup.py install * grid folder * test from grid package * setup files * minor fix * run test using coverage * foo commit * Update to latest PySyft/State and fix typo #555 * Dockerfile and sh updated * the funcionality of grid.py has been moved to __main__.py * version * syft version fixed * description modified * grid.py is in this __main__ * Fix formatting Co-authored-by: iamtrask <liamtrask@gmail.com> Co-authored-by: Bobby Wagner <raw141@case.edu> Co-authored-by: Marianne Linhares Monteiro <mariannelinharesm@gmail.com> Co-authored-by: Jon Lebensold <jon@lebensold.ca> Co-authored-by: Jose A. Corbacho <mccorby@gmail.com> Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> Co-authored-by: Ionésio Junior <ionesio@lsd.ufcg.edu.br> Co-authored-by: Amit Rastogi <amitrastogi2206@gmail.com> Co-authored-by: Vova Manannikov <12518480+vvmnnnkv@users.noreply.github.com> Co-authored-by: Prince Kumar Maurya <prince06.1996@rediffmail.com> Co-authored-by: Tallal Javed <Tallal.javed@outlook.com> Co-authored-by: midokura-silvia <43242089+midokura-silvia@users.noreply.github.com> Co-authored-by: Daniele Gadler <daniele.gadler@yahoo.it> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> Co-authored-by: Héricles Emanuel <hericles@lsd.ufcg.edu.br> Co-authored-by: João Lucas <joao.felipe@ccc.ufcg.edu.br> Co-authored-by: José Benardi de Souza Nunes <benardinunes@gmail.com> Co-authored-by: filippo balicchia <fbalicchia@gmail.com> Co-authored-by: Arjit Chauhan <arjitc12@gmail.com> Co-authored-by: Marcus Vinicius <marcus.costa@ccc.ufcg.edu.br> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: codebecker <chrisbeckerbw@gmail.com> Co-authored-by: George-Cristian Muraru <george.muraru@stud.acs.upb.ro> Co-authored-by: George-Cristian Muraru <murarugeorgec@gmail.com> Co-authored-by: Matthias Lau <lau.matthias@gmail.com> Co-authored-by: systemshift <42102034+systemshift@users.noreply.github.com> Co-authored-by: Igor <sheleztt@gmail.com> Co-authored-by: Ramjee <gantir@users.noreply.github.com> Co-authored-by: Ayoub Benaissa <ayouben9@gmail.com> Co-authored-by: chilang <chilang@gmail.com> Co-authored-by: Madeleine Shang <shang.madeleine@gmail.com> Co-authored-by: Javan Lacerda <javan.lacerda@ccc.ufcg.edu.br> Co-authored-by: Jeferson Ferreira <jefersonnpn@gmail.com> Co-authored-by: maddie_shang <mshang@tableau.com> Co-authored-by: Will <will.badr@gmail.com> Co-authored-by: kasztanka <m.lipowska@onet.pl> Co-authored-by: Patrick Cason <me@patrickcason.com>
Benardi
added a commit
that referenced
this pull request
May 12, 2020
* pip-dep folder & requirements_dev.txt * reqs moved to pip-dep folder * Makefiled updated to pip-dep folder * reqs moved to pip-dep folder * git action updated to pip-dep folder * Update run-tests.yml * initial toy websocket based example * removed pycharm files * added support files * Added example client/worker * renamed classes * intermediate commit * file running * initial grid support * added worker discovery to flask server * changed session for sqlalchemy datbase * added known workers to GridClient * checkpoint * initialized grid client and app * clean dev * reinit * a few edits * Added destroy() * added notebook * Added warning * updated warning * updated warning * updated warning * updated warning * Link PySyft in README file (#193) * small bugfix * Fixing heroku check (#194) * fixing heroku check * fixing version check for heroku * syft update * trask_grid -> dev * Fix #196 (#198) * Clean up (#201) * Clean up app.py * Create utils and clean up init * Clean up client * Clean up: part 2 (#212) * Clean up app.py * Create utils and clean up init * Clean up client * Fix typo * Create dev_user and force params * Add verify identity * Fix local redis url * cd to grid folder before checkout dev * Fix verify identity * Adding tests and how to execute them * Rename: tests -> test * Remove force (not working as expected) * Delete test_local_grid_client.py * Moving SocketIO Client and Server Worker from PySyft to Grid (#218) * fixing broken test * moving socketIO workers to grid project * Move socketio (#219) * fixing broken test * moving socketIO workers to grid project * fixing tests * Pre-commit, tests, contributors guideline (#220) * Pre-commit, tests, contributors guideline * PySyft -> Grid * Add Makefile * Examples to run with Android app (#221) * Examples to run with Android app * Fix imports * Add grid websocket approach (#217) * ADD async websocket grid app * ADD sync websocket client class * Update requirements/ change app * Rename App * Change /identity response * Deployment of websocket application * Unify address parameters at WebsocketGridClient * Little change at Procfile to deploy app with flask socket worker config * Update websocket app requirements.txt * Add -k flag at Procfile * Add eventlet worker to manage network app concurrency * [Refactor] Rename flask dir to rest_api * Add WebsocketGridClient class at grid module * ADD binascii operations * Change websocket application * Update requirements.txt * Add jupyter notebook example * Remove var type at docstrings * Create LICENSE * Add python-socketio at requirements list (#231) * Run Black (#233) * Run Black * Add coverage to requirements * Add syft to requirements * Add binder, build status, slack and FOSSA tags (#234) * Update requirements.txt (#237) * Tensor persist (#235) * fixing bug where localhost redis path not good * wip for getting postgres with a schema * moving socketIO workers to grid project * started getting flask integration test working and postgres migration * adding migration * fixing broken test * moving socketIO workers to grid project * fixing tests * got worker -> worker_objects relation working * removing whitespace * removing dead code * format fix * adding flask_sqlalchemy * adding flask_migrate * removing redundant import * unnecessary for now * commented for now * cleaning up imports * removing dead code * cleaning up more imports * added a description * Fix #238 (#242) * Grid communication (#245) * Add communication between grid nodes * Add grid at requirements.txt * Set is_client_worker flag to false / remove auto verbose mode * Run black format tool * Fix websocket app grid dependency at requirements.txt * Change log/debug message at connect grid node callback * Remove know_nodes dict * Performance Optimization between Grid nodes communication (#249) * Performance Optmization * Add notebook to measure performance before/after remove sleep procedure * Remove duplicated notebook * Add reference to Pysyft tutorial * added check for windows for tmp folder deletion (#250) * added check for windows so that tmp folder is deleted only for non-windows platform in case grid already exists * added check for verbose to output command execution status * Add ConnectionError Exception (#252) * Add ConnectionError Exception at WebsocketGrid Client * Fix code style pattern * removed flake8 docstrings from pytest in travis so travis could run (#257) * Add architecture diagram (#254) * Add architecture diagram * Improvements to the diagram * Fix notebook after changes in serde (#255) * rename tutorial file for windows compatibility (#261) Can't checkout file on windows because of ":" in the filename. Also, to be consistent with pysyft tutorials naming style: https://github.com/OpenMined/PySyft/tree/dev/examples/tutorials * REFACTORY Grid Node Application (#264) * Heroku script for creating app instance (#263) * Tensor persist router (#248) * fixing bug where localhost redis path not good * wip for getting postgres with a schema * moving socketIO workers to grid project * started getting flask integration test working and postgres migration * adding migration * fixing broken test * moving socketIO workers to grid project * fixing tests * got worker -> worker_objects relation working * removing whitespace * removing dead code * format fix * adding flask_sqlalchemy * adding flask_migrate * removing redundant import * unnecessary for now * commented for now * cleaning up imports * removing dead code * cleaning up more imports * added a description * starting flask api tests * adding persistence support for grid with unit tests * adding notes * removing unnecessary imports * removing hidden file * ignoring .DS_Store * formatting * renaming test * renaming test * better assertions * adding verbose mode * adding a workbook * green tests * reformatting * changing heroku script to use postgres * adding Flask-Migrate dependency * adding requirements * moving postgres rest api into the app folder * moving create_app * heroku app * reqs * adjusting module imports so that tests and heroku script will both work * fix import * back again * adding migration command * formatting * fix test coverage * formatting * Grid gateway (#267) * Create a initial version of grid gateway component * ADD remotely search method * ADD FederatedClient inheritance at GridNodeClient * ADD node/gateway dockerfiles + docker-compose config * ADD Search / Join Gateway routes * ADD Grid Network class * ADD /search route at grid node * ADD sy.Message() at search method * FIX Code Style * FIX route conflict * REMOVE gateway unused dependencies * FIX bug at GridNetwork search method * UPDATE README.md * UPDATE README.md * ADD new notebooks * FIX merge review requests * Change to Grid@dev at requirements.txt * run black tool * Update websocket_client.py * Update websocket_client.py * FIX REST_API_TEST (Update to sy.Message structure) * FIX connect_grid_node BUG / Run black tool * ADD WebsocketClient/Grid tests * ADD flask_cors package at requirements file * ADD eventlet package at requirements * ADD docker documentation reference at README.md file * FIX time interval at websocket/grid tests * UPDATE README.md / Decrease time interval between connections * FIX GridNetwork connection bug * REFACTORY tests * run black tool * REMOVE SocketIOAPITest class (change all test cases to pytest design) * REMOVE time.sleep() from socketio test module * run black tool * REMOVE stress test * Revert "REMOVE stress test" This reverts commit 5e330fb051dd7f4845d0f5ab5ca4baf25f3cbb3b. * Remove parameters from connect_nodes test * FIX test_connect_nodes * Update requirement files - Fix a stable version to syft lib - Remove redundant dependencies - Add requests lib at gateway requirements * Change Dockerfile's base image - python-slim base image is more lighter than ubuntu image * ADD new environment variable at gateway compose configs - Now, the gateway component will receive the secret key by environment variable * Revert changes at requirements.txt (websocket_app) - Download syft/torch packages directly from pip spend much disk space. - We have limited space on heroku platform (500 MB) * [FIX] Problem to finish grid node processes after finish test cases * run black tool * ADD time interval between websocket tests - Limited hardware resources to test environment * ADD time interval between node connections * Review requests * Deploy module (#273) * ADD deploy module - ADD BaseDeployment class (abstrac class used to initialize generic attributes) - ADD NodeDeployment class (class used to perform grid node applications deployment) * Update "Launch Websocket Grid Node" notebook - Now, perform grid node app deployment with deploy module * FIX REST_API_TEST (Update to sy.Message structure) * Sync with syft repository - In the last syft commit, the AbstractTensor class was moved to generic module. * Update requirements.txt list - Install the last version of syft library. * UPDATE requirements.txt - Add egg-name to syft package * Revert "Sync with syft repository" - This reverts commit d4c8b4f777b06438e6740392cc901f6b75fb72f0. - These changes have not yet been applied in the syft release version. * Revert changes on requirements.txt * Review requests * RENAME o -> output * FIX undefined variable name * FIX wrong var name * FIX undefined var * RENAME component -> base_deployment source file * Check if gateway_url is not None before using it + fix tests (#276) * Check if gateway_url is not None before using it * import AbstractTensor from syft.generic.tensor * Update package imports (#275) * Updated README.md (#278) * Update README.md With dev branch tutorials and projects page. * Update README.md * Launch on Heroku Update (#280) * __init__.py: - Fixed a typo in execute_command(command) - Added documentation to launch_on_heroku function - Removed unessacery if from launch_on_heroku utils.py: - Added documentation * Updated Part1 example notebook * Update __init__.py * Update utils.py * ran black for formatting on grid * Grid Node Persistence (#277) * ADD SQLAlchemy to grid websocket application - Adapt Jon's persistence features to websocket context. * Check if gateway_url is not None before using it + fix tests (#276) * Check if gateway_url is not None before using it * import AbstractTensor from syft.generic.tensor * Update package imports (#275) * Implement persistence with SQL Alchemy - Refactory SQLAlchemy Models - Remove set ID websocket event - Add recover_worker / snapshot methods * Revert set-grid-id endpoint - To the future, we'll remove this endpoint (This is not the purpose of this pull request). - Each grid node must initialize with its own id, an external component cannot change it. * MOVE db module to main package * run black tool * Snapshot method performance optmization - Before, for every command, we deleted ALL items registered on database and added ALL items again. - Now, for every command, we add/remove the difference between current and last snapshot. * RENAME db_module -> persistence * ADD doc comments * Updated README.md (#278) * Update README.md With dev branch tutorials and projects page. * Update README.md * FIX review requests * Refactor tests (#283) * IDs are lowercase and use a less common gateway port * Create folders * Clean up grid search tests * Update docker environment (#286) * Fix #271 and #287 (#285) * Fix #271 * Increase sleep time * Undo changes in test_api_socketio * Rename test * decrease sleep time * Add test for connect_all_node and fix #287 * Host model on grid node (#272) * Create port argument for websocket_app * Websocket Client inherits from Client * Create serve model route * Create experimental notebook * Return proper response * Create route for model prediction * Create _send_get * Create run_inference function * Rename model_name -> model_id and list models * Move model related methods to client * Update notebook * Clean up warnings * Create unhexlify argument * Increase ping timeout and ping interval * Support plans and jit models * Delete 2. Host a model on a grid node.ipynb * Delete Host a model on a grid node.ipynb * Organize examples folder * Improve tutorials * Move tutorials from experimental * Clean up encoding * Re-run notebook * Add tests * Update client.py * ADD support to models on Grid Gateway (#291) * Increase Grid Network support to models - ADD /choose-model-host route at gateway - ADD /search-model route at gateway - ADD /search-available-models route at gateway - ADD /search-available-tags route at gateway - ADD /dataset-tags route at websocket app - Create a new notebook * ADD unit tests (host jit model / query_model) * Improve gateway's index page * Add random.sample at choose hosts endpoint * Return the first match_node * Revert "Improve gateway's index page" This reverts commit 83af10108bcdb8e50d9a4684d9a04d87faaad734. * Solve review requests - Change list objects to set objects - Fix typo - Set ConnectionError Exception explicitly - Move n_replica's var to gateway.py * ADD host/query plans test * run black tool * Solve review requests - Remove if at gateway.py - ADD unit tests to /search-available-tags, /search-available-models endpoints * ADD run_inference method at grid - ADD run_inference method - Update Notebook - Update tests * Change host_model -> serve_model * Update Notebook * UPDATE grid tests * HOST GPT-2 (#292) * ADD new Notebook * Fix Notebook * Fix typo * ADD requests-toolbelt dependencies * ADD Text generation at Notebook * Solve review requests * Update requirements.txt (#294) - Fixes deployment issues on Heroku with size - Added Postgres driver * Host and Query an Encrypted model (#288) * Create 3. Host and Query an Encrypted model * Rename 3. Host and Query an Encrypted model to 3. Host and Query an Encrypted model.ipynb * Change model to convnet * Use a plan * Support fetching a plan * Remove unused notebook cells * Restart notebook to show that fetch is working properly * Use convnet * Get pointers instead of getting objects * Update for changes in fetch * Add id to model * Support private model hosting and querying * Create a new notebook * Fix notebook * Update reference to correct attribute * Clean up and add comments * Unify serve model and document get_private_model * Add persistent models (#282) * Create port argument for websocket_app * Websocket Client inherits from Client * Create serve model route * Create experimental notebook * Return proper response * Create route for model prediction * Create _send_get * Create run_inference function * Rename model_name -> model_id and list models * Move model related methods to client * Update notebook * Clean up warnings * Create unhexlify argument * Increase ping timeout and ping interval * Persistant Model Storage - model_manager.py for saving, deleting, listing and in short handling all the models - routes.py: updated for model manager and created a new route of deleting a model - jupyter notebook example updated with the delete function and some commenting to make it easy to understand - client.py updated with the delete_model route * A sample pickle model save file. * Support plans and jit models * Delete 2. Host a model on a grid node.ipynb * Delete Host a model on a grid node.ipynb * Organize examples folder * - Added database * Improve tutorials * Move tutorials from experimental * Clean up encoding * Re-run notebook * Add tests * deleted model files. As they are shifted to db. * Improved Comments - Improved comments - Removed conflicts - Improved exception handling * model cache - removed model list from routes - Adding model cache to model_manager. * removed todo * ran black. * Style Fixes - Style fixes - model_manager now returns dicts - Updated client to handle new requests - Success status added in model manager and flask responses. * Merged with dev - Merged with dev - Ran black - Removed unnecessary prints * fixed bug in check_if_model_exists - fixed bug in check_if_model_exists. - better comments and return dicts. * Use LargeBinary instead of BYTEA * Rename MLModel -> TorchModel * Style fixes and fix clear cache * Add missing json loads * Add delete operation * Remove experimental notebook * Add missing json.loads * Remove unused variable and add comment * Add TorchTensor * Create local_worker var * Create file with local worker utilities * Cache real object + fix plans db support * Use local worker utilities * Add tests for deleting a model * Use encoding * Fix GPT-2 demo * Rename fetch_plan to fetch_plan_reference * Fix imports * Use sy.messaging * Fix Message * Fix imports * Create notebook 2.2 * Delete 3. Host and Query an Encrypted model.ipynb * Avoid error during snapshot * Host and Query encrypted models options * Remove fetch_plan_reference and demo 3.3 * Remove get_private_model and demo 3.1 * Rename notebook * Fix make_helper jit support * Add db support for encrypted models * Use local_worker utilities * Clean up model manager * Relaocate comments * Fix issues in Syft for now * Fix torchscript db issue * Add test test_host_encrypted_model * Fix test_host_encrypted_model * Replace 2.2 notebook * Add test_run_encrypted_model * Clean up websocket app and gateway app starting scripts (#296) * Fix typo (tst_config -> test_config) * Use parser and os environ vars as default * Fix import order and typos * Define warning * Fix tests * Update requirements.txt (#297) - fetching syft from dev branch directly. * Clean up host model API (#298) * Add new columns to TorchModel * Add allow_model_copy and allow_run_inference args * Snapshot shouldn't include model objects * Fix tests * Create get_serialized_model_with_id * Create get_model route * Create get_model_copy * Create global variables for storing error messages * Rename allow_model_copy -> allow_get_model_copy * Fix typo * Implement serve_encrypted_model * Improve comments * Add tests for new arguments * Fix test * Standardize results * Update tests * Fix small bug * Improve delete error messages * Fix typo * Fix test * Extend syft with a friendly api * Better argument names * Try to fix tests * Try to fix tests * Fix tests * Added TODO * Update notebooks * download -> download_model * Wrap host/inference encrypted models (#300) * Wrap host/inference encrypted models - Wrap encrypted serve_model / run_inference - ADD /search-encrypted-models at websocket app - ADD /search-encrypted-model / /choose-encrypted-host at grid gateway * Fix typo * ADD encrypted serve_model/run_inference test * FIX serve_model/run_inference test * Fix typo / ADD constant value * Fix review requests * Update requirements.txt (#297) - fetching syft from dev branch directly. * Run black tool * ADD comment about eventlet money_patch * ADD new notebook * Fix heroku ws app (#302) * Improve 3.2 * Fix heroku app * Get rid of main * Update notebook * Fix websocket_app.py * args.gateway_url -> gateway_url * Change Model ids (db.Integer -> db.BigInteger) * Update notebook * Remove outdated comment * Add Introduction Notebooks (#303) * Start What's Grid Notebook * Update and rename 1 - What's Grid?.ipynb to Part 0 - Introduction to Grid * Create Part 1 - Launch a Grid Node Locally.ipynb * Update and rename Part 1 - Launch a Grid Node on Heroku.ipynb to Part 2 - Launch a Grid Node on Heroku.ipynb * Update Notebook (Launch a Grid Node on Heroku) * ADD .ipynb extension at notebook * FIX Launch a Grid Node on Heroku notebook * ADD deployment support to grid gateway * Fix Procfile / gateway.py * Fix websocket_app bug * ADD heroku_gateway deployment module * Improve notebooks * Update demo (#305) * Improve 3.2 * Update demos * Update notebooks * Add note about checking tutorials * Remove Serving models * Review examples (#307) * improve readability of tutorials (#308) * Fix tutorial serving querying (#310) * Fix for psycopg2 package installation on Linux * Fixed torch 1.0.1 requirement for Linux and Windows * Fixed imports' names for the gp2_utils file and added explanation on how to correctly parse the HMNIST dataset * Reverted changes related to dependencies for torch * Reverted psycopg fix * small improvements in readme and tutorial (#313) * small improvements in readme and tutorial * correcting typos and adding some explanations * Fix typos in tutorials (#309) * Fix typos * Update Part 1 - Launch a Grid Node Locally.ipynb * Fix typo * Fix test generation * General fixes / improvements * ADD Gateway persistence (#314) * ADD Gateway persistence * ADD Persistence at Gateway Deployment * ADD Notebook * Update 4. Introducing the Grid Network.ipynb (#316) * Web Index page update (#317) * web initial * model list being fetched from server * finalized web index page * ran black * changed gradient background - to cover. * added comment for model.pop explanation. * fixed typo - removed test endpoint. * ran black * ADD webpage section (#322) * Fix for psycopg2 and torch 1.0.1 package installation on Linux (#301) * Fix for psycopg2 package installation on Linux * Fixed torch 1.0.1 requirement for Linux and Windows * Fixed PyTorch package version for Python 3.7 * Added python runtime environment for Heroku * Disallow cross-origin resource sharing to webpage routes (#323) * Add README files for app folders (#324) * Improve 3.2 * Add README.md files in app * Add grid logo to Readme (#329) * Update README.md added logo and changed grid to PyGrid. * Update README.md * Update model_manager.py (#331) fixes #318 - in case where save_model gives an integrity error * Common Install Issues Section (#334) * Common Installatoin Issues Section fixes #306 * Update README.md * Add api documentation (#328) * Verify download permissions at /get_model/<model_id> route - If someone performs a request to this endpoint outside of grid lib context, they probably will get the model (Security Problem). - Now, /get_model/<model_id> endpoint returns 403 (Forbidden) if not allowed to download. - Now, /get_model/<model_id> endpoint returns 404 (Not Found) If not found at the grid node. * CHANGE /model/<model_id> request method GET -> POST We can forward HTTP body content at GET requests, but it's a bad practice. We should use POST/PUT methods if we want to forward important data between REST API. * Change response error code at /serve-model - Replace 500 Internal Server Error -> 409 Conflict 5xx rest codes are used to represent server errors. 4xx rest codes are used to represent client errors. In this case, our response error message will be sended when we have a already registered model This error was be created by the client side, not server. * ADD Grid Node REST API documentation * Fix markdown * Fix grid_node_rest_api document * ADD Grid Node Websocket API Documentation * ADD gateway REST API * FIX gateway persistence bug * FIX Notebooks examples * FIX api documents * FIX gateway_api document * Merge grid clients (#333) * Merge WebsocketGridClient and GridClient - Move all GridClient's methods to WebsocketGridClient * Remove GridClient class - Remove GridClient reference. - Remove remove GridClient tests. * FIX flake error * Merge Grid Node Apps (#335) * Remove unused apps * Remove outdated/unused tests * Remove notebook(copy) * Socket Grid Node (#338) * [REFACTORY] Grid Node App - Change SocketIO protocol to Pure Websockets Protocol (Now we can send/receive messages from syft.js) - Move some HTTP routes to socket API - Allow to receive/send binary contents (Perfomance Improvement) - Add event_routes source file (handle socket requests) * [REFACTORY] Grid Lib - Remove unused/outdated source file (grid/cliemt.py) - Extends syft.WebsocketClientWorker - Change HTTP node requests to socket node requests - Delete unused/outdated methods * [CHANGES] Tests - Change conftest (update to test new grid_nodes) - Rename test_api_socketio -> test_api_socket - test_api_sockets changes (update to test new grid_nodes) - test_api_grid changes (update to test new grid lib) * ADD gevent/gevent-websocket at requirements.txt * Updaate Procfile * Update requirements dependencies * Fix for psycopg2 on gateway (#344) Replaced the `psycopg2` package with `psycopg2-binary` in the gateway/requirements.txt file. * Update travis badge (#346) * Update travis badge Replaced `https://travis-ci.org/OpenMined/Grid` to `https://travis-ci.org/OpenMined/PyGrid` * Skip test_grid_search Add @pytest.mark.skip decorator at test_grid_search unit test * fixing formatting error * ordered requirements to install grid (#347) * Update the PyGrid components web user interface (#348) * update titles from 'Grid' to 'PyGrid' in web interface * create web interface for pygrid network based on node interface * Add Issue Templates (#349) * create issue template: feature request * create issue template: bug report * Add sphinx config to build docs (#351) * Add sphinx config to build docs Edit conf.py to set rtcat theme. Set napoleon to allow Google style pydocs. * Add documentation build to travis CI verifications (#353) * add new parameter to docker-compose (#340) * add dataset tags table and fix ui grid (#357) * Update index.html (#358) * Update template.css (#361) * Add setup.cfg/pbr and make package Ready-to-Publish (#362) * Add pbr and setup.cfg * Edit setup.cfg and README for valid pypi package Edit invalid setup fields. Set README image to hosted files at Github. * Add Pull Request template (#364) * Add Pull Request template * Edit PR template to clarify requests * Edit PR template to remove typo * Update PyGrid-Arch Image (#365) * Add PyGrid-Arch image * Update PyGrid Architecture Image on README * Authentication (#342) * ADD Auth module at grid lib * Update grid __init__.py * ADD Auth module at Grid Node App - Create Session Repository - Create UserSession - Socket Session Auth wrapper - Authenticate socket route * ADD authenticate method * SKIP test_host_plans * Fix problem with _load_preset_credentials * ADD flask_login lib * Handle Multiple Private/Public Workers - Now we're able to connect using different private workers at the same node. - Map private workers by user attestation. - We're able to connect with local private/public workers. - We're able connect private/public workers with remote nodes. * Remove unused source file * ADD docstrings * Remove custom search method at grid node * Remove search method at grid.websocket_client * Remove unused imports * Update Authenticate event response * Move model_manager source file to persistence module * Fix /workers/ endpoint * ADD default admin infos * Update auth module * Fix search method * RENAME create_standard_auth -> register_new_credential * RENAME load_auth_credentials -> load_credentials * ADD docstrings at auth module * MOVE 'set_database_config' to persistence module * REMOVE local_worker_utils source file * ADD docstrings at PyGrid App * REMOVE outdated functions 'launch_on_heroku' * FIX/ADD Docstrings at PyGrid lib * CREATE grid_codes source files * REFACTORY events - Create events module at PyGrid Node App - events.py -> events/__init__.py - event_routes -> syft_events.py, model_events.py, control_events.py * Update grid_codes.py * Skip Plans CI tests - PySyft last update crash Plans CI tests/ grid functions, it will fixed in another PR. * Fix flake/black code style errors * Add entrypoint.sh (#368) * route to delete a grid node (#367) * route to delete a grid node * fixing routes with black * runing black at manager file * Bump werkzeug from 0.15.2 to 0.15.3 in /gateway (#369) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.15.2 to 0.15.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/0.15.2...0.15.3) Signed-off-by: dependabot[bot] <support@github.com> * Adding MNIST Federated Example (#366) * Adding MNIST Federated Example * Cleaning notebooks outputs/Adding cuda support to mnist federated example * Adding empty folder to download datasets / Changing datasets path * Updating files permissions * Bump werkzeug from 0.15.2 to 0.15.3 in /app/websocket (#370) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 0.15.2 to 0.15.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/master/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/0.15.2...0.15.3) Signed-off-by: dependabot[bot] <support@github.com> * Edit docs to serve it remotely (#375) * Mock deps for autodoc * Replace autodoc for autoapi Set autoapi to averts heavy dependencies * Edit sphinx conf to enforce theme css * Edit sphinx to include generated api documentation * Conform code to black standards * Fix file permissions (#376) * updating websocket route event (#378) * Update local examples to conform to grid module (#382) * Fix serde and logging imports in Android worker example notebook (#380) * Move documentation/ files into docs hierarchy (#384) * Updated torch version for worker container (#386) * Replace Grid with PyGrid (#387) * Add spam Example (#388) * Add files from SPAM Folder * Spam Populate Grid Network Notebook * Spam Train Model * Black reformat for Spam example * Spam fix typo (#391) * Fix bug on remote inference (#392) * updating websocket route event * undo commit * fixin response message on websocket app * Start Android example server on `0.0.0.0` to accept all connections (#394) Change address to '0.0.0.0' to accept all connections * Download model bug (#396) * updating websocket route event * undo commit * changing way to get serialized model * Change PySyft branch from dev -> master (#397) PySyft no longer hosts its latest code in the dev branch. We push straight to master. * Remove unused/outdated import (#398) * Refactory gateway Routes (#403) * Gateway Routes Refactory - Decrease Cyclomatic complexity. - Avoid/Handle Internal Server Errors * ADD INVALID_JSON_FORMAT_MESSAGE constant * Fix the websocket app procfile module to websocket_app (#404) * Rename folders used in heroku deploy scripts from grid to PyGrid (#406) * Add missing proto dependency to the websocket app (#405) * Process tensor request (#399) * Remove unused/outdated import * Handle GetNotPermittedError and save tensor request * Add CI tests * Update PySyft repository reference * Fix requirements.txt * Fix .travis install session * Add egg information to requirements.txt (#409) On my machine (OSX 10.14.6 running python 3.6), while I was able to install requirements.txt using 'pip install -r requirements.txt', I was not ablee to do so by running 'pip install .' as the README directs. After some googling, adding egg information to the github imports in requirements.txt allowed me to do so. I don't claim to be an expert on this issue, but it worked for me. * remove_Connecting_Locally.ipynb (#414) * Decode error (#400) * updating websocket route event * undo commit * adding try catch to decode bytes that utf-8 cant decode * [FIX] serve_models / run_encrypted_inferences on plans. (#420) * [GridNode] Fix host/run_inference on plans * [PyGrid Lib] Update serve_model method. * [CI Tests] Allow / Update tests using plans. * Fix encrypted inference route * [CI Tests] Allow/Update host/run_inference/run_encrypted_inferences at grid tests. * Fix requirements.txt for heroku The syft_proto is currently failing to install on Heroku with the HerokuNodeDeployment script. Trying to fix it * Use PyPI version of syft_proto * Bugfix * Bugfix * Fix app requirements.txt (#422) * [Bug FIX] Standard tensors also have the get method, even without children. (#425) * Fix requirements hierarchy (#423) * Fix app requirements.txt * Remove proto dependency from requirements.txt, fix dependency hierarchy (it'll avoid package incompatibilities between syft/pygrid and syft-proto) * Update docker-compose.yml (#424) * Update docke-compose / README * Fix README.md * Correct Fell->Feel type in the README. (#426) There is a noticeable typo. Please accept this patch to fix it. * Kubernetes Config file (#415) * docker files * Updated commands formatting * Added services, removed unnecessary entries from deployment * connecting to the local gateway service * moved files to appropriate base folders * rationalised the label names, updated image of node to default * add gateway overlays for dev and prod. * basic kustomize working for gateway * started using config maps for env vars * used configmaps for node * deploy script * updated service to be available externally * Fixed the deploy to proper shape * added todo, renamed k8s to readme * updated readme.ms * Update todo.md * added docker build grid-gateway, grid-node * removed changes for to travis.yml * update docker image * support for multiple nodes * support for multiple nodes in prod * support for multiple nodes in test Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Updated k8s Readme (#427) * docker files * Updated commands formatting * Added services, removed unnecessary entries from deployment * connecting to the local gateway service * moved files to appropriate base folders * rationalised the label names, updated image of node to default * add gateway overlays for dev and prod. * basic kustomize working for gateway * started using config maps for env vars * used configmaps for node * deploy script * updated service to be available externally * Fixed the deploy to proper shape * added todo, renamed k8s to readme * updated readme.ms * Update todo.md * added docker build grid-gateway, grid-node * removed changes for to travis.yml * update docker image * support for multiple nodes * support for multiple nodes in prod * support for multiple nodes in test * Updated the docker build image name * updated Readme.md with kubernetes, docker image name Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Add mpc flag / column for encrypted models (#428) * Replacing Grid Node's PyGrid/PySyft dependencies (#429) * REFACTORY: replace lib (grid -> syft) at grid node app. * REFACTORY: replace lib (grid -> syft) at CI tests. * Update syft temporarily * remove grid imports * ADD private_grid unit tests * Update requirement files * Remove unused imports * Add import warning Co-authored-by: Andrew Trask <liamtrask@gmail.com> * [Refactory] Grid node's persistence module (#432) * Remove SQLAlchemy models / source files * Refactory Persistence module * Fix syft remote search * Update websocketapp / syft_events - Update websocketapp to receive redis url by args / environment var - Update syft_events to import object_storage and use it to recover worker * Fix get model from cache * Remove model instance checkup * Remove save/get plan states * Update model events * Fix/Update persistence module * Update CI tests configs * Update gitignore / add database source file * Run black tool * Update requirements.txt * Update requirements.txt * Change event error message * storage cache encapsulation * Fix/Update routes * Update database server * Replace literal strings by codes * Replace literal strings by syft.codes * Add docstrings * Updating grid status and name at grid app (#443) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * Grid Standardization (#447) * starting gateway implementation over websockets * create ws blueprint * fix some imports * Fix flake/black issues * Add ws events sketch * - Create Scope Class - Create Scope Controller - implement get-protocol message * create codes for response * Add socket handler to manage connections * Fix get-protocol endpoint glitches * Implement webrtc routes * fix black errors * Add docstrings * create test file for gateway websockets api * add socket client for test mocking * update gateway app runner on conftest and fix socket ping endpoint * update requirements.txt with aiounittest * add socket_ping test * add some tests for gateway websockets * fix black issues Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Setup GitHub Actions Workflow (#448) * setup github actions workflow * exclude syft * exclude syft * update syft repo * skip some tests and update syft origin * remove travis and update build status badge (#449) * Consistent Model Hosts (#450) * Search nodes for ones already hosting the model when choosing model host * Add a test for model host consistency * Improve k8s documentation (#452) * Update requirements.txt (#442) Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Create Federated Learning Processes (#458) * create a draft of a federated learning process class * create a controller to manage federated learning processes * fix black issues * update classname and make client_protocols a default arg * Use docker port binding instead of host networking (#461) Solves #460 * working docker compose after removing extra '/' in REDISCLOUD_URL. Both notebooks https://github.com/OpenMined/PySyft/blob/master/examples/tutorials/grid/federated_learning/mnist/ should works now Need to add 127.0.0.1 -> host.docker.internal to /etc/hosts to work around Docker for Mac not supporting network_mode: 'host' (#462) Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Create Workers (#470) * create possible user class * create worker concept * WIP: fl_download_model_endpoint (#471) * Change PySyft Reference (#474) * Add flask ws worker as a Docker ENTRYPOINT (#475) * Remove outdated grid library source code (#476) * Remove outdated grid lib * Remove library documentation * remove unused file * Remove grid lib config files * Update Git CI tests * Remove outdated tests using grid library * Update README.md (#477) * Update README * Add PyGrid image * Remove image from README.md * remove readthedocs file * Updating Launch examples (#478) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * adding route to download protocol * getting the issue done * fixing merge to PyGrid dev * download protocol http endpoint * download protocol http endpoint * download protocol http endpoint * finishing issue #446 * deleting federated route * updating examples how launch grid * changing python version * changing python version * changing python version part3 * changing python version part3 * changing python version part4 Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Sketch Events for Mobile/Web Federated Learning (#469) * Refactory pygrid codes * Remove scope events * Add fl_events sketch * Update events module * Skip scope tests * Fix code style * Remove unused codes * Fix expected JSON fields * Add model_id / remaining_time * Fix code style * Add FederatedLearningCycle class * Fix imports * Minor fixes * Fix Flake8 issues * Fix black tool issues * Support Cycle sequences * Rename FLPController -> FLController * Fix import FLPController -> FLController * Protocol download endpoint (#480) * updating websocket route event * undo commit * adding node name and changing status * status now changes if node is connected * doc change * deleting debug log * adding route to download protocol * getting the issue done * fixing merge to PyGrid dev * download protocol http endpoint * download protocol http endpoint * download protocol http endpoint * finishing issue #446 * deleting federated route * updating examples how launch grid * changing python version * changing python version * changing python version part3 * changing python version part3 * changing python version part4 * adding new route to get protocols * changing branch * fixing black * changing protocols to cliente_protocols Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Setup GH Actions Workflow for Automated Docker build (#484) * Create docker-build.yml * Update run-tests.yml * Delete docker-build.yml * Create docker-build.yml * Create docker-build.yml * Update docker-build.yml * Update docker-build.yml * Update docker-build.yml * remove conditional * update grid node path * Update CI badges in README (#485) * Create ansible playbook to deploy the pygrid (#472) * Create ansible playbook to deploy the pygrid * Refactor playbook/README and add destroyer playbook * update README with destroy playbook usage * remove unecessary vars reference Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Update README and Contributing guidelines (#488) * Update links and numbers * Update CONTRIBUTING.md * Update requirements.txt (#492) * Update Docker Publish Workflow (#487) * Update docker-build.yml * Rename Job Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fl report endpoint (#489) * report endpoint Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * pr for feedback fl_impFLcycles - use server config (#479) * WIP: fl_download_model_endpoint * mock up use of server_config params for #438-2 * mock up use of server_config params for #438-2 * math stuff * wip: model in place * wip: model in place * wip: model in place * wip: model in place * wip: model in place * comment code * comment code * comment code Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Add FL http cycle request endpoint (#497) * [WIP] add federated/cycle-request endpoint to gateway routes * [WIP] enable worker to get cycles and update cycle-request endpoint * no changes. just to avoid overwritten by merge * (fix) remove get_cycles function * PyGrid Gateway Persistence (#481) * Initial commit * Fix tablename * Update models.py Init Schemas * Update models.py Fix typos. * Update models.py ADD tablenames / fix docstrings * Update models.py Add One-To-Many/One-To-One relationships. * Create __init__.py * Update requirements.txt Add PySyft as a gateway requirement. * Update models.py Finish model schemas * Create database_tests.py Add database unit tests * Update models.py Add syft import * Fix code style * Rename module Rename module : persistence -> storage * Update __init__.py replace import: persistence -> storage * Update routes.py replace import persistence -> storage * Fix code style * Rename database CI tests * Create warehouse.py Add generic data warehouse class. * Update controller.py Rename dict (workers -> cache) / Add storage attribute * Update requirements.txt Fix PySyft reference * ADD host_federated_training event ADD host_federated_training WS event * Update controller.py Add an worker database interface * Update controller.py Add worker database interface * Update models.py Fix docstrings/ add new field at protocol/plan * Update warehouse.py Fix multi args operators * Update fl_events.py Update events * Update controller.py Fix datetime * Update models.py Add ModelCheckPoint.alias * Update controller.py Fix argument operators * Move database_tests ADD database tests to CI unit tests * Update FL Process * Update models/fl_controller * Update controller.py Add PyGrid Exception handling * Create exceptions.py Create PyGrid's Exceptions * Update fl_events.py Improve response messages / remove not necessary print * Update codes.py Add new codes (accepted, rejected, success) * Update routes.py - Add PyGrid Exception Handling - Refactory federated/get-model - Refactory federated/get-protocol - Refactory federated/cycle-request - Remove static messages * Update models.py - Update docstrings - Uncomment syft library * Update controller.py - Add PyGrid's Exception Handling - Add useful get methods * Add syft library * Update fl_events.py - Add unhexlify - deserialize model * Update controller.py - Fix create_process parameter - Remove hard coded cycle's version * Update test_gateway_sockets.py Add test_fl_process * Update test_gateway_sockets - Fix fl_process CI test Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Update gateway Dockerfile (#499) - Fix zstd issue during building docker image Co-authored-by: José Benardi de Souza Nunes <benardinunes@gmail.com> * Remove json.dumps from host_traning_model request (#501) * Remove scope (#505) * Remove Scope module - Legacy Code * Update worker.py Fix typos * Delete webrtc_events.py Remove scope events * Update __init__.py Remove webrtc_events import * Add federated/get-plan route (#482) * Add federated/get-plan route 'receive_operations_as' parameter being ignored * Fix var plan naming * Fix code formatting Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fix request cycle (#503) * Update models.py - Change model id type to auto-increment integer - Rename checkpoints back reference * Update controller.py - Remove pre-defined id - Save initial model checkpoint * Update fl_events.py - Remove model deserialization * Update test_database.py Update Model tests * Fix FL process references - Now, We can retrieve a fl process by using the name defined at client_config * Update test_gateway_sockets.py -Fix FL Process test * Update routes.py - Fix little bug (inverted parameter order) * Update exceptions.py - Add new exception (FLProcessConflict) * Update controller.py - Check if FLProcess / Version already exists - Fix request_cycle bugs * Update models - Add name/version fields at FLProcess * Update warehouse.py - Fix .contains method * Run black tool * Update models.py Add version as a parameter in the FL Process model * Update controller.py - Uncomment model_checkpoints registration - Add version as a parameter during FL Process queries * Update FL unit test - Add version during cycle-request * Update controller.py - Add get_plan method * Update Search method (#508) - Remove custom search method. - Remove syft.search overwrite * PR ready: average a plan (#498) * averages a plan based on diffs Co-authored-by: maddie_shang <mshang@tableau.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * Add build context in docker-compose.yml (#510) Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * skip fl tests (#512) * skip fl tests * code formatting * FL Process API fixes (#509) * Fix FL host request; make get-model and get-plan to return binary * formatting with black Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * remove scope tests (#511) * Fix FL tests (#513) * Fix FL tests * Black * Update fl_events.py (#515) * Update fl_events.py Add field "type" on WS responses * Update test_gateway_sockets.py Update tests * Update test_gateway_sockets.py Fix fl tests * add dummy auth endpoint to unblock (#516) * add dummy auth endpoint to unblock * add dummy auth endpoint to unblock Co-authored-by: maddie_shang <mshang@tableau.com> * Update FL report, make diff avg async (#518) * Update FL report, make diff avg async * Fix formatting * Fix unit tests * Fix formatting * Update HTTP responses (#520) * Update routes.py Fix HTTP requests issues. * Update routes.py Code style fix * Update request cycle (#521) Closes #514 * Update controller.py - Add model name and version to rejected responses. - Remove timeout field if it's the last cycle. * Update test_gateway_sockets.py - Add check on version field during request-cycle test * Update requirements.txt (#522) Updating flask and requests to match with syft 0.2.4 * Fix bug in docker-compose.yml file. (#524) When DATABASE_URL changed to REDISCLOUD_URL three slashes were mistakenly kept. * Create gunicorn dev conf (#523) * create gunicorn dev conf * code formatting with black Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Create docs with running instructions for grid gateway and node (#525) * Update README.md * Create README.md * Add running instructions * Update README.md * Add docker section * Update README.md * Update README.md * Updating requirements.txt package versions (#529) * Updating requirements.txt package versions Updating the gateway requirements to match the versions required by PySyft. * removing flaskSocket from requirements.txt FlaskSocket IO is no longer required * auth working (JWT + HSA + RSA) (#517) * auth working * fix placeholder imports * black clashes with pre-commit hook * wip: debug no module named jwt * fix req * RSA and HSA in * RSA and HSA in * add reqs * remove prints * fix db stub * remove prints * clean up * clean up Co-authored-by: maddie_shang <mshang@tableau.com> * Update PyGrid tutorials (#528) * remove heroku tutorials * update intro to pygrid * update intro tutorials * rename tutorials * delete android examples * fix black issues * DELETE outdated tutorials Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> * Fix Docker network interface (#533) * Fix Docker network interface * Update README.md * Move Grid Node to a different repository (#531) * MOVE Grid Node Move Grid Node source code to another repository : https://github.com/OpenMined/GridNode * Update CI tests Comment Grid Nodes tests * FIX code style * Fix code style * Delete unused tests * Remove unused conftest settings * Move PyGrid app source files * Update requirements.txt - Add CI dependencies at requirements.txt file * Update requirements.txt * [REFACTORY] RENAME Rename app gateway -> grid * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update docker-build.yml (#534) Update the "workdir" reference * Connection speed test (#526) * Update routes.py Add connection_speed_test * Update fl_events.py Fix Max Cycle Limit * Update routes.py Fix connection_speed_route * Update routes.py fix typo * Update exceptions.py Add MaxCycleLimiteExceeded Exception * Update routes.py Update test_download / test_upload methods * Update requirements.txt Add requests-toolbelt dependency * Remove unused folder/files * Add requests_toolbelt dependency * Update routes.py - Increase the data sample size to 64 MBytes Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> * [REFACTORING] Improving PyGrid's code quality (#535) * Rename storage -> models - Rename module (storage -> models) - Split daatabase models into different source files - Move db singleton instance to models module * Update models module * Update CI test_database * Fix host_federated_training test * Update import references * REFACTORY fl_controller - Move complete_cycle, average_plan_diffs to cycle_manager - Fix worker.is_eligible method - Update report event * Update models module * Update Grid Architecture - Split database models into different modules * Fix small bugs * Fix flake issues * Update imports * Create routes module Split route source file into different source files. - federated - network - general * Update federated.py * Rename module - node_routing -> network * Update routes.federated.py - Replace mockup vars to database commands described by @Prtfw * Trim Plan to torchscript or "list of ops" variants and serve them (#536) * Convert Plan to torchscript and serve it * Host request error handling * Update to follow PySyft changes * Fix cycle manager, make protocol optional in cycle request * Update requirements.txt Co-authored-by: Patrick Cason <me@patrickcason.com> * Update docker-compose.yml (#540) Make it easier to build new gateway image * pre-commit in reqs_dev * fix merge errors * requirements.txt deleted * requirements moved to pip-dep * delete app folder because moved to grid folder * python setup.py install * grid folder * test from grid package * setup files * minor fix * run test using coverage * foo commit * Update to latest PySyft/State and fix typo #555 * Dockerfile and sh updated * the funcionality of grid.py has been moved to __main__.py * version * syft version fixed * description modified * grid.py is in this __main__ * Fix formatting Co-authored-by: iamtrask <liamtrask@gmail.com> Co-authored-by: Bobby Wagner <raw141@case.edu> Co-authored-by: Marianne Linhares Monteiro <mariannelinharesm@gmail.com> Co-authored-by: Jon Lebensold <jon@lebensold.ca> Co-authored-by: Jose A. Corbacho <mccorby@gmail.com> Co-authored-by: Ionésio Junior <ionesiojr@gmail.com> Co-authored-by: Ionésio Junior <ionesio@lsd.ufcg.edu.br> Co-authored-by: Amit Rastogi <amitrastogi2206@gmail.com> Co-authored-by: Vova Manannikov <12518480+vvmnnnkv@users.noreply.github.com> Co-authored-by: Prince Kumar Maurya <prince06.1996@rediffmail.com> Co-authored-by: Tallal Javed <Tallal.javed@outlook.com> Co-authored-by: midokura-silvia <43242089+midokura-silvia@users.noreply.github.com> Co-authored-by: Daniele Gadler <daniele.gadler@yahoo.it> Co-authored-by: Théo Ryffel <theo.leffyr@gmail.com> Co-authored-by: Héricles Emanuel <hericles.me@gmail.com> Co-authored-by: Héricles Emanuel <hericles@lsd.ufcg.edu.br> Co-authored-by: João Lucas <joao.felipe@ccc.ufcg.edu.br> Co-authored-by: José Benardi de Souza Nunes <benardinunes@gmail.com> Co-authored-by: filippo balicchia <fbalicchia@gmail.com> Co-authored-by: Arjit Chauhan <arjitc12@gmail.com> Co-authored-by: Marcus Vinicius <marcus.costa@ccc.ufcg.edu.br> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Karl Higley <kmhigley@gmail.com> Co-authored-by: codebecker <chrisbeckerbw@gmail.com> Co-authored-by: George-Cristian Muraru <george.muraru@stud.acs.upb.ro> Co-authored-by: George-Cristian Muraru <murarugeorgec@gmail.com> Co-authored-by: Matthias Lau <lau.matthias@gmail.com> Co-authored-by: systemshift <42102034+systemshift@users.noreply.github.com> Co-authored-by: Igor <sheleztt@gmail.com> Co-authored-by: Ramjee <gantir@users.noreply.github.com> Co-authored-by: Ayoub Benaissa <ayouben9@gmail.com> Co-authored-by: chilang <chilang@gmail.com> Co-authored-by: Madeleine Shang <shang.madeleine@gmail.com> Co-authored-by: Javan Lacerda <javan.lacerda@ccc.ufcg.edu.br> Co-authored-by: Jeferson Ferreira <jefersonnpn@gmail.com> Co-authored-by: maddie_shang <mshang@tableau.com> Co-authored-by: Will <will.badr@gmail.com> Co-authored-by: kasztanka <m.lipowska@onet.pl> Co-authored-by: Patrick Cason <me@patrickcason.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
There's a lot of outdated grid tutorials since the
gridlibrary was moved to PySyft. This PR aims to update them.List of changes
/androidexamplesType of change
Please mark options that are relevant.
Checklist: