docs: Document synchronization mutex usage. (#7006)

Signed-off-by: Josh Carp <jm.carp@gmail.com>

docs/synchronization.md

@@ -9,7 +9,9 @@ Synchronization enables users to limit the parallel execution of certain workflo
 templates within a workflow without having to restrict others.
 
 Users can create multiple synchronization configurations in the `ConfigMap` that can be referred to 
-from a workflow or template within a workflow.
+from a workflow or template within a workflow. Alternatively, users can
+configure a mutex to prevent concurrent execution of templates or
+workflows using the same mutex.
 
 For example:
 ```yaml
@@ -27,7 +29,7 @@ Workflow-level synchronization limits parallel execution of the workflow if work
 In this example, Workflow refers `workflow` synchronization key which is configured as rate limit 1, 
 so only one workflow instance will be executed at given time even multiple workflows created. 
 
-example:
+Using a semaphore configured by a `ConfigMap`:
 
 ```yaml
 apiVersion: argoproj.io/v1alpha1
@@ -49,12 +51,32 @@ spec:
       args: ["hello world"]
 ```
 
+Using a mutex:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  generateName: synchronization-wf-level-
+spec:
+  entrypoint: whalesay
+  synchronization:
+    mutex:
+      name: workflow
+  templates:
+  - name: whalesay
+    container:
+      image: docker/whalesay:latest
+      command: [cowsay]
+      args: ["hello world"]
+```
+
 ### Template-level Synchronization
 Template-level synchronization limits parallel execution of the template across workflows, if template have same synchronization reference. 
 In this example, `acquire-lock` template has synchronization reference of `template` key which is configured as rate limit 2, 
 so, two instance of templates will be executed at given time even multiple step/task with in workflow or different workflow refers same template. 
 
-example:
+Using a semaphore configured by a `ConfigMap`:
 
 ```yaml
 apiVersion: argoproj.io/v1alpha1
@@ -85,9 +107,42 @@ spec:
       command: [sh, -c]
       args: ["sleep 10; echo acquired lock"]
 ```
+
+Using a mutex:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  generateName: synchronization-tmpl-level-
+spec:
+  entrypoint: synchronization-tmpl-level-example
+  templates:
+  - name: synchronization-tmpl-level-example
+    steps:
+    - - name: synchronization-acquire-lock
+        template: acquire-lock
+        arguments:
+          parameters:
+          - name: seconds
+            value: "{{item}}"
+        withParam: '["1","2","3","4","5"]'
+
+  - name: acquire-lock
+    synchronization:
+      mutex:
+        name: template
+    container:
+      image: alpine:latest
+      command: [sh, -c]
+      args: ["sleep 10; echo acquired lock"]
+```
+
 Examples:
-1. [Workflow level](https://github.com/argoproj/argo-workflows/blob/master/examples/synchronization-wf-level.yaml)
-2. [Step level](https://github.com/argoproj/argo-workflows/blob/master/examples/synchronization-tmpl-level.yaml)
+1. [Workflow level semaphore](https://github.com/argoproj/argo-workflows/blob/master/examples/synchronization-wf-level.yaml)
+1. [Workflow level mutex](https://github.com/argoproj/argo-workflows/blob/master/examples/synchronization-mutex-wf-level.yaml)
+1. [Step level semaphore](https://github.com/argoproj/argo-workflows/blob/master/examples/synchronization-tmpl-level.yaml)
+1. [Step level mutex](https://github.com/argoproj/argo-workflows/blob/master/examples/synchronization-mutex-tmpl-level.yaml)
 
 ### Other Parallelism support:
 In addition to this synchronization, the workflow controller supports a parallelism setting that applies to all workflows