Correctness and documentation of the samples #85
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidmrdavid
changed the title
davidmrdavid
changed the title
Right. Python Durable doesn’t use any of the new features in Durable Functions v2 yet. Even when v2 is available when bundles v2 is published, all tooling will still default to using bundles v1 for some time, so using v1 should result in a better dev experience in the short term. Definitely up for debate though but that’s my current thinking. |
priyaananthasankar
approved these changes
Mar 24, 2020
davidmrdavid
changed the title
|
The latest commit includes changes for the tensorflow example as well, meaning that we can be confident that our samples work with Python 3.6 - 3.7 I have also added types and docstring comments. I would appreciate a new round of reviews before merging. Thanks! |
davidmrdavid
changed the title
|
Added preliminary READMEs for |
|
I have also renamed some of the function names. |
…nts now shows no warnings!
|
Latest commit disables aiohttp's input validation, which is necessary for |
This was referenced Mar 25, 2020
|
Merging as I've got Anthony's "spoken" approval already. |
priyaananthasankar
added a commit
that referenced
this pull request
Mar 30, 2020
* Docstring cont Orchestration state docstrings added * Docstring cont. Doc strings for the retry options * run flake8 only in azure/folder * demo sample for function chaining with docstring and json changes * pep8 naming add some pep8 naming to the exposed APIs * Ignore hardcoded string tests Need to decide whether we keep them or refactor to use ContextBuilder Tests are redundant with orchestrator tests as is * Work through regression bugs after renaming * Remove redundant tests * Docstrings for several api level objects OchrestrationState, Task, TaskSet, clean up some excess docstrings * Package docstrings docstrings defined for actions, history, utils and tasks packages * task_utility docstrings * ActivityType docstrings Contains refactoring to conform to python naming conventions to * Doc strings for Actions Call Activity and CallActivity with Retry * Doc Strings History Included refactor for enum naming conventions in python * Final docstring bits docstrings for tasks and json utilities * add waitforexternalevent basic files * fix bugs to make waitForExternalEvent working * add waitforexternalevent samples * remove explicit binding_info imp defined in conftest as a fixture, explicit import not required. * demo sample for function chaining with docstring and json changes * demo sample for function chaining with docstring and json changes * added dev pipeline status in README * minor fixes(variable name, delete comment) * flake8 fixes * add docstrings * implement task_any function * change pip install library and add docstrings to samples * change pip install library and add docstrings to samples * unittest for waitforexternalevents * fix bugs after merging dev * fix flake8 * Base implementation of tests test initial call test get activity and build of task set * parrot values success * test full complete flow * test failed scenario add missing bits in task_all to account for a failed task * docstring to numpy format * minor changes (rename, remove logging) * unittest for task_any, added tasks_test_utils * add class __eq__ function for Waitforexternalevent actions * add samples readme doc * fix flake8 * Refactoring HistoryEvent Get rid of all of those hardcoded property value references * add docstrings for HistoryEvent class * Refactor json conversion refactor the classes that are parsing json strings to remove all of the hardcoded property names also allows for the classes dangle additional attributes that may be present but not explicitly used for construction * simple Fan out fan in sample * Fix flake errors * Remove local debugging bits * remove state in task_any * add handle faulted task_any case +unittest * Undo De Morgan's Law cause it's really hard to read * replace filters with list comprehension more concise/readable * Add documentation for tracking API implementation * move datetime format string to azure package * replace filter with list comprehension more concise method * remove extra zimezone from format causing parsing warnings * Push context initialization our of handle method have this dependency injected instead of built within * able to pass in tasksets to task_any and task_all * update unittest for adding timestamp to taskset, add unittest for passing taskset to task_any as args * fix bugs in task_all(when all tasks fail), and fix unittest for that scenario * fix flake8 * test from orchestrator level(draft) * Remove IFunctionContext abstraction unneeded layer of abstraction with a DUCK typing language like Python * Starting of schema validation bits * createCheckStatusResponse() * wire up schema validation into the orchestrator tests * Test commit * fix flake 8 issues * fix pytest, remove task_any_tests from orchestrator level * fix flake8 * implement raise_event api, fix docstring * add unittest, create separate methods, fix naming style, handle placeholder in url validation * Fan Out Fan In sample uses tensorflow to predict whether images from bing are dogs or cats * fix flake8 * add aiohttp to requirement.txt and setup.py * add async await to start_new and raise_event api, flake8 * update api_implementation_status * fix variable naming style in docstring * update sample for external events, update readme in sample/external_events * Refactoring and docstrings removed the get generation data function. With the configurability, we can increase the volume of images to simulate the high CPU scenario. No need to fake it now. refactored some of the larger, multi task functions to create better readability added some docstrings on the functions to aid in understanding what is going on without the visual aid of powerpoint * Continue as new implementation * new_guid implementation * Fix flake8 issues * update sample to simple version * add func.httpresponse in durableorchestratorClient.py * update docstring for createcheckstatusresponse api * fix flake8 * update sample for create_check_status_response fixes * update pytest for the changes in createcheckstatusresponse api, add azure-function to requirement.txt for pytest * Implementation of call_http * Remove traceback print statement * remove url validator * Updates from demo feedback rename guid to uuid attach additional context attributes to an attribute of the durable context, not directly to the durable context * Fix flake8 * copy paste error giving a more descriptive orchestrator name * remove abcd from sample * Update API_IMPLEMENTATION_STATUS.md Sprint 7 update * Unit tests for call_http found a couple of bugs with these * merged shervyna changes * Fix the squiggles mostly formatting/inspection surpession * Move iAction moving and renaming IAction to Action * Update actions to implement action base class * remove the squiggles some minor formatting updates * Fix format of docstring * update to use rpc_base_url for start new and raise event urls * minor updates docstring on new function add Action to __init__ for actions module * flake8 bits * Add action_type property of base class * update docstrings per convention * docstring convention updates made sure docstrings conformed to the same format and indentions * Base class implementations for get_status * refactor massive string into a Dict[str, Any] object makes for a more readable bit of code * Move duplicate test rpc url constants to centralized location * adding updated contributing and getting started guides * get_status implementation includes refactoring of aiohttp calls to be able to test the rest of the bits without this server requirement tests for GetStatusOptions included base test for get_status_success * test for DurableOrchestrationStatus parsing * Add non ok message tests * add raises exception test * get_status_by and get_status_all implementation * flake8 fixes * fix docstring for get_status_by parameters copy paste error * addig nox to the project - updated requirements file * Updated noxfile to include flake, move getting started to investigations folder * typo in the commit from Jscript to python * PR recommendations remove PyCharm # noinspection tags move get and post async functions to util module replace abc123 stand-in test value with a uuid stand-in * add docstrings for new public functions * Continued Client API implementation purge_instance_history purge_instance_history_by terminate wait_for_completion_or_create_check_status_response * bug fixed for retry function. * refactor tests remove the redundant function declarations * unit tests for purge history and terminate * unit tests for wait or create check response includes bug fixes found from tests * update implementation status with features included in branch * update name match the name of the class under test * change to use asynio sleep don't want a blocking sleep call here * Use azure-functions>=1.2.0 * Make DurableOrchestrationContext available for import * Update start_new type hints to return str * Add get_input to DurableOrchestrationContext * Allow passing OrchestrationContext to create - maintain backwards compatibility when JSON body is passed * Change str to Awaitable[str] * Add get_input tests, get_input always returns str * Fix flake8 errors * add furl to setup.py * Fix merge error * # This is a combination of 3 commits. (#58) Fix samples to use the latest func.OrchestrationContext annotation Use new syntax to replace main endpoint Fix old python_durable_bindings samples * Replace badge with the new Azure Functions Python pipeline * edited contributor guide * furl is required for this project - added it to the requirements.doc * fixed pytest-asyncio in requirements.txt * fixed pytest-asyncio in requirements.txt * fixed pytest-asyncio in requirements.txt * Remove static method tag (#72) Remove static method tag (#72): residual bits left over from when a part of the client class. Causing failures in current state. Also removing unnecessary wrapper of http helper calls * Remove furl from RpcManagementOptions only durable class that is requiring this package, and is only using it to construct a url query string. Lighter requirements list to just use string joining * release artifacts * Added DateUtil and Furl to install_requires * changelog and setup.py changes * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * tested version of pipeline yaml * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipeline * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * setup.py CD testing * setup.py CD testing * PyPI testing * added info logging to the external event sample * Correctness and documentation of the samples (#85) * validated all samples except tensorflow * deleted non-samples, tested tensorflow sample, added types and comments * new readmes for fan-out-fan-in and function-chaining * renamed some functions so they are easier to reason about * disable aiohttp's input validation... potentially unsafe. ExternalEvents now shows no warnings! * Fixed Contributor Guide Issues * Fixed Contributor Guide Issues * Input value not required for start_new * input values not required for start_new * Add limitations, link to quickstart, update samples * Update example * Add link * Update words * Add versions * Update links Co-authored-by: Shawn Gaul <shgaul@microsoft.com> Co-authored-by: Shervyna Ruan <HsuanWei.Ruan@microsoft.com> Co-authored-by: Shawn Gaul <shawn.gaul@gmail.com> Co-authored-by: asedighi <sediga@alum.rpi.edu> Co-authored-by: Anthony <antchu@microsoft.com> Co-authored-by: Anthony Chu <anthony@anthonychu.ca> Co-authored-by: Hanzhang Zeng (Roger) <48038149+Hazhzeng@users.noreply.github.com> Co-authored-by: Hanzhang Zeng (Roger) <hazeng@microsoft.com> Co-authored-by: David Justo <david.justo.1996@gmail.com>
priyaananthasankar
added a commit
that referenced
this pull request
Jun 18, 2020
* task_utility docstrings * ActivityType docstrings Contains refactoring to conform to python naming conventions to * Doc strings for Actions Call Activity and CallActivity with Retry * Doc Strings History Included refactor for enum naming conventions in python * Final docstring bits docstrings for tasks and json utilities * add waitforexternalevent basic files * fix bugs to make waitForExternalEvent working * add waitforexternalevent samples * remove explicit binding_info imp defined in conftest as a fixture, explicit import not required. * demo sample for function chaining with docstring and json changes * demo sample for function chaining with docstring and json changes * added dev pipeline status in README * minor fixes(variable name, delete comment) * flake8 fixes * add docstrings * implement task_any function * change pip install library and add docstrings to samples * change pip install library and add docstrings to samples * unittest for waitforexternalevents * fix bugs after merging dev * fix flake8 * Base implementation of tests test initial call test get activity and build of task set * parrot values success * test full complete flow * test failed scenario add missing bits in task_all to account for a failed task * docstring to numpy format * minor changes (rename, remove logging) * unittest for task_any, added tasks_test_utils * add class __eq__ function for Waitforexternalevent actions * add samples readme doc * fix flake8 * Refactoring HistoryEvent Get rid of all of those hardcoded property value references * add docstrings for HistoryEvent class * Refactor json conversion refactor the classes that are parsing json strings to remove all of the hardcoded property names also allows for the classes dangle additional attributes that may be present but not explicitly used for construction * simple Fan out fan in sample * Fix flake errors * Remove local debugging bits * remove state in task_any * add handle faulted task_any case +unittest * Undo De Morgan's Law cause it's really hard to read * replace filters with list comprehension more concise/readable * Add documentation for tracking API implementation * move datetime format string to azure package * replace filter with list comprehension more concise method * remove extra zimezone from format causing parsing warnings * Push context initialization our of handle method have this dependency injected instead of built within * able to pass in tasksets to task_any and task_all * update unittest for adding timestamp to taskset, add unittest for passing taskset to task_any as args * fix bugs in task_all(when all tasks fail), and fix unittest for that scenario * fix flake8 * test from orchestrator level(draft) * Remove IFunctionContext abstraction unneeded layer of abstraction with a DUCK typing language like Python * Starting of schema validation bits * createCheckStatusResponse() * wire up schema validation into the orchestrator tests * Test commit * fix flake 8 issues * fix pytest, remove task_any_tests from orchestrator level * fix flake8 * implement raise_event api, fix docstring * add unittest, create separate methods, fix naming style, handle placeholder in url validation * Fan Out Fan In sample uses tensorflow to predict whether images from bing are dogs or cats * fix flake8 * add aiohttp to requirement.txt and setup.py * add async await to start_new and raise_event api, flake8 * update api_implementation_status * fix variable naming style in docstring * update sample for external events, update readme in sample/external_events * Refactoring and docstrings removed the get generation data function. With the configurability, we can increase the volume of images to simulate the high CPU scenario. No need to fake it now. refactored some of the larger, multi task functions to create better readability added some docstrings on the functions to aid in understanding what is going on without the visual aid of powerpoint * Continue as new implementation * new_guid implementation * Fix flake8 issues * update sample to simple version * add func.httpresponse in durableorchestratorClient.py * update docstring for createcheckstatusresponse api * fix flake8 * update sample for create_check_status_response fixes * update pytest for the changes in createcheckstatusresponse api, add azure-function to requirement.txt for pytest * Implementation of call_http * Remove traceback print statement * remove url validator * Updates from demo feedback rename guid to uuid attach additional context attributes to an attribute of the durable context, not directly to the durable context * Fix flake8 * copy paste error giving a more descriptive orchestrator name * remove abcd from sample * Update API_IMPLEMENTATION_STATUS.md Sprint 7 update * Unit tests for call_http found a couple of bugs with these * merged shervyna changes * Fix the squiggles mostly formatting/inspection surpession * Move iAction moving and renaming IAction to Action * Update actions to implement action base class * remove the squiggles some minor formatting updates * Fix format of docstring * update to use rpc_base_url for start new and raise event urls * minor updates docstring on new function add Action to __init__ for actions module * flake8 bits * Add action_type property of base class * update docstrings per convention * docstring convention updates made sure docstrings conformed to the same format and indentions * Base class implementations for get_status * refactor massive string into a Dict[str, Any] object makes for a more readable bit of code * Move duplicate test rpc url constants to centralized location * adding updated contributing and getting started guides * get_status implementation includes refactoring of aiohttp calls to be able to test the rest of the bits without this server requirement tests for GetStatusOptions included base test for get_status_success * test for DurableOrchestrationStatus parsing * Add non ok message tests * add raises exception test * get_status_by and get_status_all implementation * flake8 fixes * fix docstring for get_status_by parameters copy paste error * addig nox to the project - updated requirements file * Updated noxfile to include flake, move getting started to investigations folder * typo in the commit from Jscript to python * PR recommendations remove PyCharm # noinspection tags move get and post async functions to util module replace abc123 stand-in test value with a uuid stand-in * add docstrings for new public functions * Continued Client API implementation purge_instance_history purge_instance_history_by terminate wait_for_completion_or_create_check_status_response * bug fixed for retry function. * refactor tests remove the redundant function declarations * unit tests for purge history and terminate * unit tests for wait or create check response includes bug fixes found from tests * update implementation status with features included in branch * update name match the name of the class under test * change to use asynio sleep don't want a blocking sleep call here * Use azure-functions>=1.2.0 * Make DurableOrchestrationContext available for import * Update start_new type hints to return str * Add get_input to DurableOrchestrationContext * Allow passing OrchestrationContext to create - maintain backwards compatibility when JSON body is passed * Change str to Awaitable[str] * Add get_input tests, get_input always returns str * Fix flake8 errors * add furl to setup.py * Fix merge error * # This is a combination of 3 commits. (#58) Fix samples to use the latest func.OrchestrationContext annotation Use new syntax to replace main endpoint Fix old python_durable_bindings samples * Replace badge with the new Azure Functions Python pipeline * edited contributor guide * furl is required for this project - added it to the requirements.doc * fixed pytest-asyncio in requirements.txt * fixed pytest-asyncio in requirements.txt * fixed pytest-asyncio in requirements.txt * Remove static method tag (#72) Remove static method tag (#72): residual bits left over from when a part of the client class. Causing failures in current state. Also removing unnecessary wrapper of http helper calls * Remove furl from RpcManagementOptions only durable class that is requiring this package, and is only using it to construct a url query string. Lighter requirements list to just use string joining * release artifacts * Added DateUtil and Furl to install_requires * changelog and setup.py changes * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * tested version of pipeline yaml * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipeline * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * setup.py CD testing * setup.py CD testing * PyPI testing * added info logging to the external event sample * Correctness and documentation of the samples (#85) * validated all samples except tensorflow * deleted non-samples, tested tensorflow sample, added types and comments * new readmes for fan-out-fan-in and function-chaining * renamed some functions so they are easier to reason about * disable aiohttp's input validation... potentially unsafe. ExternalEvents now shows no warnings! * Fixed Contributor Guide Issues * Fixed Contributor Guide Issues * Input value not required for start_new * input values not required for start_new * Add limitations, link to quickstart, update samples * Update example * Add link * Update words * Add versions * Update links * versioning-via-tags is enabled (#101) * updated readme for external events (#103) * Readme file for FanOutFanIn (#104) * Readme file for FanOutFanIn * Readme file for FanOutFanIn * Readme file for FanOutFanIn * Add activity trigger return type sample (#105) * Add sample for activity trigger type checks * Remove extensions.csproj Co-authored-by: Priya Ananthasankar <prananth@microsoft.com> * misc contributions (#108) * Set Custom Status (#110) * custom status * custom status * custom status * custom status * custom status * custom status * custom status * custom status * set custom status rework comments * Enabled nox to recognize docstring formatting errors (#122) * enables trivial orchestrators * added flake8-docstring, fixed arg to flake8 ,fixed formatting on noxfile * removed accidental commit from other PR * enables trivial orchestrators (#121) * removed grpc folder (#120) * [WIP] Semi-automatic type-serialization (#109) * Promote to master (#99) (#126) Co-authored-by: Shawn Gaul <shgaul@microsoft.com> Co-authored-by: Shervyna Ruan <HsuanWei.Ruan@microsoft.com> Co-authored-by: Shawn Gaul <shawn.gaul@gmail.com> Co-authored-by: asedighi <sediga@alum.rpi.edu> Co-authored-by: Anthony <antchu@microsoft.com> Co-authored-by: Anthony Chu <anthony@anthonychu.ca> Co-authored-by: Hanzhang Zeng (Roger) <48038149+Hazhzeng@users.noreply.github.com> Co-authored-by: Hanzhang Zeng (Roger) <hazeng@microsoft.com> Co-authored-by: David Justo <david.justo.1996@gmail.com> Co-authored-by: Priya Ananthasankar <prananth@microsoft.com> Co-authored-by: Shawn Gaul <shgaul@microsoft.com> Co-authored-by: Shervyna Ruan <HsuanWei.Ruan@microsoft.com> Co-authored-by: Shawn Gaul <shawn.gaul@gmail.com> Co-authored-by: asedighi <sediga@alum.rpi.edu> Co-authored-by: Anthony <antchu@microsoft.com> Co-authored-by: Anthony Chu <anthony@anthonychu.ca> Co-authored-by: Hanzhang Zeng (Roger) <48038149+Hazhzeng@users.noreply.github.com> Co-authored-by: Hanzhang Zeng (Roger) <hazeng@microsoft.com> * Promote to master (#99) (#127) * Monitoring Feature (#119) * updated CHANGELOG (#128) * Changelog includes links (#129) * updated CHANGELOG * improved changelog descriptions w/ issue links Co-authored-by: Shawn Gaul <shgaul@microsoft.com> Co-authored-by: Shervyna Ruan <HsuanWei.Ruan@microsoft.com> Co-authored-by: Priya Ananthasankar <prananth@microsoft.com> Co-authored-by: Shawn Gaul <shawn.gaul@gmail.com> Co-authored-by: asedighi <sediga@alum.rpi.edu> Co-authored-by: Anthony <antchu@microsoft.com> Co-authored-by: Anthony Chu <anthony@anthonychu.ca> Co-authored-by: Hanzhang Zeng (Roger) <48038149+Hazhzeng@users.noreply.github.com> Co-authored-by: Hanzhang Zeng (Roger) <hazeng@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update: No longer work in progress, feel free to merge
This PR validates that our Python samples execute correctly under Python 3.8. The only sample I was not able to test yet is the Tensorflow one because it depends on Python 3.6 and our development container does not allow me to
apt installPython 3.6.The changes in this PR include:
extensions.csprojfileshost.jsonfiles to use bundlesfunction.jsonfiles to use theorchestrationClientinstead ofdurableClient, because the latter depends on Functions V2 and it appears that bundles are not supported for V2 yet? Please confirmI would appreciate a skim over the changes.