[Feature Request] Investigate/document how to use PyCharm debugger with workflows

[cretz]

Describe the solution you'd like

May be similar to #238 but maybe not (that's more about the sandbox).

It seems when trying to debug via replayer in PyCharm, some things break that work just fine when not using PyCharm's debugger. Investigate what breaks and why. This may be due to some patching/expectations PyCharm has around asyncio that don't hold true for custom asyncio implementations. There's another issue at https://youtrack.jetbrains.com/issue/PY-64542/Asyncio-support-for-the-debugger-doesnt-work-with-pytest-asyncio.

We need to document how exactly to use breakpoints in workflows from this IDE.

[mnichols]

@cretz here is my first crack at such a doc:

Python developers must take into consideration both the sandbox and the Workflow Task Timeout for any Debugger interactions. Here are some tips to help you "step through" Workflow Code with a Debugger.

Debugging Workflow Code

Finding problems within complex Workflow code might be aided by stepping through
the code as you normally would to inspect variable state, step through execution paths, etc.

Workflow code executed by a Worker does so within the bounds of a Workflow Task which is
itself governed by a Timeout.

• The maximum amount of time a Workflow Task can be configured to take before Failed is 120 seconds.
• The default Timeout is 10 seconds.
• That means you have a maximum of two minutes to diagnose your code before the current Task will be failed and rescheduled.
  You'll see a flood of error messages in your console related to these Task issues. This may not be enough time to perform your diagnosis.

So how can I step through my Workflow Code??

You can get around this Task boundary by executing your code within a Replayer.
This is done by passing Workflow Execution history into the Replayer while the debugger is attached to step through the code.
Executed this way, there is no Task to interfere with your diagnosis.

PyCharm users might experience issues with asyncio debugging.
If you are having difficulty debugging with PyCharm, you might consider the steps in this ticket
to solve it. The python.debug.asyncio.repl can be toggled within PyCharm by:

1. Use Shift + Shift (Search Everywhere)
2. In the popup type: Registry and press Enter
3. Find "Registry" in the list of results and click on it.
4. In the new popup find python.debug.asyncio.repl line and check (enable) or uncheck (disable) the respective checkbox
5. Press Close.
6. Restart the IDE.

Here are a couple of approaches to debug Temporal Workflows with a Debugger:

The Debug Two-Step

This approach involves running first without the Debugger attached to generate Workflow Execution histories.
Subsequently, these histories can be used in the second run with the Debugger attached.
This approach can be handy if you already have Execution histories you want to verify that failed, for example, in production.

First run (without Debugger attached)

1. Execute the Workflow(s) and keep the Temporal Server running (if testing locally)

Second run (with Debugger attached)

1. Set appropriate breakpoints in Workflow code.
2. Fetch the related Workflow Execution History(ies) and store in a variable
3. Pass the history into a Replayer with the following properties set:

replayer = Replayer(
    workflows=[MyWorkflow], 
    debug_mode=True,
    namespace='debug',
    workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner())

4. Profit.

The Debug One-Shot

1. Set appropriate breakpoints in Workflow code, but this time add a Condition of `workflow.info().namespace == 'debug'.
2. Run the test with Debugger attached that:
   1. Executes the Workflow(s)
   2. Fetches the related Workflow Execution History(ies) and store in a variable
   3. Passes the history into a Replayer with the following properties set:

replayer = Replayer(
    workflows=[MyWorkflow], 
    debug_mode=True,
    namespace='debug',
    workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner())

3. Profit.