run_workflow_template only passes variables when prompt on launch selected in Workflow Job Template #622

adworjan · 2023-11-27T16:58:18Z

Please confirm the following

I agree to follow this project's code of conduct.
I have checked the current issues for duplicates.
I understand that ansible-rulebook is open source software provided for free and that I might not receive a timely response.

Bug Summary

When using the run_workflow_template and a survey, the workflow job only receives the variables when prompt on launch is also selected for extra vars. This is not an issue when directly calling the API, but is an issue when using the run_workflow_template action

Environment

ansible-rulebook-1.0.3
Ansible Automation Platform Controller 4.4.8

Steps to reproduce

Create a workflow job template
Create and enable a survey (do NOT check prompt on launch for extra variables in the WF)
Create a rulebook with the run_workflow_job template and the job_args to populate that variable
Trigger the rulebook

Actual results

If the survey is set to required, it fails saying that variable is needed to start

2023-11-27 16:00:40,040 - ansible_rulebook.rule_set_runner - INFO - action args: {'name': 'Automated Response Node Exporter Down EDA', 'organization': 'Security', 'job_args': {'extra_vars': {'vm_name': 'rhel8.shadowman.dev'}}}

2023-11-27 16:00:40,041 - ansible_rulebook.action.run_workflow_template - INFO - running workflow template: Automated Response Node Exporter Down EDA, organization: Security

2023-11-27 16:00:40,041 - ansible_rulebook.action.run_workflow_template - INFO - ruleset: Automatic Remediation from Alert Manager, rule Prometheus Node Exporter Down

2023-11-27 16:00:40,292 - ansible_rulebook.job_template_runner - WARNING - Workflow template Automated Response Node Exporter Down EDA does not accept limit, removing it

2023-11-27 16:00:40,292 - ansible_rulebook.job_template_runner - WARNING - Workflow template Automated Response Node Exporter Down EDA does not accept extra vars, removing it

2023-11-27 16:00:40,524 - ansible_rulebook.job_template_runner - ERROR - Error connecting to controller 400, message='Bad Request', url=URL('https://tower.shadowman.dev/api/v2/workflow_job_templates/612/launch/')

2023-11-27 16:00:40,524 - ansible_rulebook.job_template_runner - ERROR - Error {'variables_needed_to_start': ["'vm_name' value missing"]}

Expected results

The job should successfully run without setting prompt on launch for extra vars since the survey exists

Additional information

No response

mkanoor · 2023-11-27T18:02:37Z

@adworjan In our testing with 4.4.0 we noticed the following error if we send extra vars into a launch of workflow template

{"extra_vars":["Variables abc are not allowed on launch. Check the Prompt on Launch setting on the Workflow Job Template to include Extra Variables."]}

Along with a bad request from the controller.

So if there is no prompt on launch enabled in the Workflow Template we won't be able to start the job.

This is different from a Job Template, a Job Template launch seems to accept extra vars even if the Prompt on Launch is not enabled.

adworjan · 2023-11-27T18:05:37Z

This is only an issue when using the run_workflow_template action. I can call the API directly without having prompt on launch checked (but with the survey enabled). Your test seems to not have a survey with variable abc set

mkanoor · 2023-11-27T18:58:54Z

When we get an event payload there is all kinds of data in it and we don't validate any of the data to see if it matches the survey spec types.

{'variables_needed_to_start': ["Value 23 for 'abc' expected to be an integer."], 'extra_vars': ['Variables hello, ansible_eda are not allowed on launch. Check the Prompt on Launch setting on the Workflow Job Template to include Extra Variables.']}

So even if we checked the prompt_on_launch for variables and survey enabled we are still going to see launch failures because of the kinds of data that we might get in the event payload. Currently EDA doesn't do anything with survey specs, we don't fetch it we don't validate if the extra vars match the name/type specified in the survey spec.

adworjan · 2023-11-27T19:07:33Z

The issue I have isn't with the survey spec type (I'm using the text type). I can send an API call to that workflow launch endpoint and the job runs with the extra var passed in. If I use the run_workflow_template it fails UNLESS I have prompt on launch checked. I'm still populating the exact same variable in the survey, but it only populates with prompt on launch checked. Every API call I make to controller works with JUST the survey enabled. This works fine in the run_job_template action, so the issue appears to be in the run_workflow_template action itself.

mkanoor · 2023-11-27T19:13:18Z

The run_workflow_template action is intentionally removing the extra_vars if the prompt on launch for extra vars is not set. The run_job_template doesn't do that. The job template launch on the controller is not as restrictive as is the case with workflow template launch. Even in the run_job_template action we have been recommending users enable the prompt on launch so you can see what is the extra vars that were sent up. #566

adworjan · 2023-11-27T19:18:14Z

Can we remove that restriction? It would make sense to provide consistency between both actions (and just like with a Job Template, I might not need the ansible_eda data, I might just want a single variable). I purposely only use surveys so users don't have to know what variables to populate AND so I don't need a separate workflow for EDA vs regular usage.

mkanoor · 2023-11-27T19:36:22Z

That restriction is in place because of the controller restrictions on workflow templates launch procedure. Without the prompt on launch the run_workflow_template will still fail because of the fact that we are always sending the matching events under the ansible_eda key. This is the extra_vars that would get sent up

{'extra_vars': {'hello': 'Fred', 'abc': 23, 'ansible_eda': {'ruleset': 'Test run workflow templates', 'rule': 'Run workflow template', 'event': {'i': 1, 'meta': {'source': {'name': 'range', 'type': 'range'}, 'received_at': '2023-11-27T19:27:04.946004Z', 'uuid': '928b170e-14d3-4c3c-ba2c-b01c54620ae1'}}}}}

The ansible_eda is an object with ruleset/rule/event. This object cannot be represented in the Survey Spec, if you represented it as a string you would get a different error

This is the controller error for that

Error {'variables_needed_to_start': ["Value OrderedDict([('ruleset', 'Test run workflow templates'), ('rule', 'Run workflow template'), ('event', OrderedDict([('i', 1), ('meta', OrderedDict([('source', OrderedDict([('name', 'range'), ('type', 'range')])), ('received_at', '2023-11-27T19:27:04.946004Z'), ('uuid', '928b170e-14d3-4c3c-ba2c-b01c54620ae1')]))]))]) for 'ansible_eda' expected to be a string."]}

So for this to work we would have to remove sending the event data in the extra vars, today it is being sent unconditionally.

If the ansible_eda is not defined in the survey spec. the error changes to

ERROR - Error {'extra_vars': ['Variables ansible_eda are not allowed on launch. Check the Prompt on Launch setting on the Workflow Job Template to include Extra Variables.']}

goldyfruit · 2023-12-11T23:31:12Z

This is different from a Job Template, a Job Template launch seems to accept extra vars even if the Prompt on Launch is not enabled.

Hi there,

"Job Template" doesn't receive as well the extra vars if Prompt on Launch is not checked, just spent the last hour trying to figure out why extra variables were not passed to the job template (as well as ansible_eda).

Enabling Prompt on Launch made my job template receives the extra vars:

{
  "which_user": "Gaetan",
  "ansible_eda": {
    "ruleset": "Demo rules with websocket as source",
    "rule": "Run job template",
    "event": {
      "type": "ovos-skill-personal.OpenVoiceOS:WhoMadeYou.intent",
      "data": {
        "utterances": [
          "who made you"
        ],
        "lang": "en-us",
        "utterance": "who made you"
      },
      "context": {
        "session": {
          "active_skills": [
            [
              "ovos-skill-personal.OpenVoiceOS",
              1702337196.0405762
            ]
          ],
          "utterance_states": {},
          "session_id": "default",
          "lang": "en-us",
          "context": {
            "timeout": 120,
            "frame_stack": []
          },
          "site_id": "unknown",
          "pipeline": [
            "converse",
            "padatious_high",
            "adapt",
            "common_qa",
            "fallback_high",
            "padatious_medium",
            "fallback_medium",
            "padatious_low",
            "fallback_low"
          ]
        },
        "lang": "en-us",
        "skill_id": "ovos-skill-personal.OpenVoiceOS"
      },
      "meta": {
        "source": {
          "name": "websocket",
          "type": "smartgic.eda.websocket"
        },
        "received_at": "2023-12-11T23:26:36.071602Z",
        "uuid": "05181bba-384f-4d33-bb2c-89eab804f0f1"
      }
    }
  }
}

mkanoor · 2023-12-12T14:53:27Z

This is different from a Job Template, a Job Template launch seems to accept extra vars even if the Prompt on Launch is not enabled.

Hi there,

"Job Template" doesn't receive as well the extra vars if Prompt on Launch is not checked, just spent the last hour trying to figure out why extra variables were not passed to the job template (as well as ansible_eda).

Enabling Prompt on Launch made my job template receives the extra vars:

{
  "which_user": "Gaetan",
  "ansible_eda": {
    "ruleset": "Demo rules with websocket as source",
    "rule": "Run job template",
    "event": {
      "type": "ovos-skill-personal.OpenVoiceOS:WhoMadeYou.intent",
      "data": {
        "utterances": [
          "who made you"
        ],
        "lang": "en-us",
        "utterance": "who made you"
      },
      "context": {
        "session": {
          "active_skills": [
            [
              "ovos-skill-personal.OpenVoiceOS",
              1702337196.0405762
            ]
          ],
          "utterance_states": {},
          "session_id": "default",
          "lang": "en-us",
          "context": {
            "timeout": 120,
            "frame_stack": []
          },
          "site_id": "unknown",
          "pipeline": [
            "converse",
            "padatious_high",
            "adapt",
            "common_qa",
            "fallback_high",
            "padatious_medium",
            "fallback_medium",
            "padatious_low",
            "fallback_low"
          ]
        },
        "lang": "en-us",
        "skill_id": "ovos-skill-personal.OpenVoiceOS"
      },
      "meta": {
        "source": {
          "name": "websocket",
          "type": "smartgic.eda.websocket"
        },
        "received_at": "2023-12-11T23:26:36.071602Z",
        "uuid": "05181bba-384f-4d33-bb2c-89eab804f0f1"
      }
    }
  }
}

@goldyfruit What I meant was Job Template launch doesn't fail if the "Prompt on Launch" is missing, it just ignores the extra vars and keeps going. In the case of Workflow Template the launch fails if we send it extra vars when Prompt on Launch is not defined.

goldyfruit · 2023-12-12T14:54:45Z

@goldyfruit What I meant was Job Template launch doesn't fail if the "Prompt on Launch" is missing, it just ignores the extra vars and keeps going. In the case of Workflow Template the launch fails if we send it extra vars when Prompt on Launch is not defined.

Thanks for the clarification.

When we unconditionally send event data to the controller for a workflow template the launch of the workflow template fails. With this option in a rulebook the user can prevent event data from being sent by setting - include_events: false The default value is true so existing rulebooks will keep sending events to controller. Fixes #622 https://issues.redhat.com/browse/AAP-13456 Root Cause Analysis: In the Controller if you are running a workflow template you cannot send event data to it unconditionally like we can do with job template. We get an error from Controller **Check the Prompt on Launch setting on the Workflow Job Template to include Extra Variables.** If a survey spec is involved then there are more constraints if there are certain parameters that are marked as required in the survey spec. So we get 2 distinct items - Prompt on Launch for Extra Variables - Survey Spec The following use cases are supported for workflow job template with and without survey spec. The use cases are based on a rulebook that looks like the following with the action being updated for every case. ``` --- - name: Test run job templates hosts: all sources: - ansible.eda.generic: payload: - name: Fred age: 25 rules: - name: "Run job template" condition: event.age == 25 action: run_workflow_template: name: workflow_mk organization: Default ``` ### **Case 1:** If your Workflow Template doesn't support Prompt on Launch for Extra Vars and a Survey Spec is not enabled ``` run_workflow_template: name: workflow_mk include_events: false organization: Default ``` ### **Case 2:** If your Workflow Template supports Prompt on Launch for Extra Vars and a Survey Spec is not enabled ``` run_workflow_template: name: workflow_mk organization: Default job_args: extra_vars: name: "{{ event.name }}" ``` The payload sent to Controller ``` name: Fred ansible_eda: ruleset: Test run job templates rule: Run job template event: meta: source: name: ansible.eda.generic type: ansible.eda.generic received_at: '2024-09-26T05:29:10.983346Z' uuid: 9570c7f8-9bc9-438f-9199-803707e31ccf name: Fred age: 25 ``` ### **Case 3:** If your Workflow Template doesn't support Prompt on Launch for Extra Vars and a Survey Spec is enabled ``` run_workflow_template: name: workflow_mk include_events: false organization: Default job_args: extra_vars: name: "{{ event.name }}" ``` The payload sent to Controller ``` name: Fred ``` ### **Case 4:** If your Workflow Template supports Prompt on Launch for Extra Vars and a Survey Spec is enabled ``` action: run_workflow_template: name: workflow_mk organization: Default job_args: extra_vars: name: "{{ event.name }}" ``` The payload sent to Controller ``` name: Fred ansible_eda: ruleset: Test run job templates rule: Run job template event: meta: source: name: ansible.eda.generic type: ansible.eda.generic received_at: '2024-09-26T07:26:42.160723Z' uuid: 46169e51-ba6f-4a55-963b-b626fff7cfdc name: Fred age: 25 ``` ### **Case 5:** Survey Spec has a required parameter and its not provided ``` action: run_workflow_template: name: workflow_mk organization: Default ``` Missing args 2024-09-26 12:59:03,738 - ansible_rulebook.job_template_runner - ERROR - Error {'variables_needed_to_start': ["'name' value missing"]} ### **Case 6:** Survey Spec has no required parameter and its not provided ``` action: run_workflow_template: name: workflow_mk organization: Default ``` The Default Value is provided by the Survey Spec and it works ``` name: fred ansible_eda: ruleset: Test run job templates rule: Run job template event: meta: source: name: ansible.eda.generic type: ansible.eda.generic received_at: '2024-09-26T07:55:15.320036Z' uuid: 68a6c25d-dcf5-4a1c-8eb2-7432376d2d88 name: Fred age: 25 ```

mkanoor mentioned this issue Sep 26, 2024

feat: option to skip including events into extra_vars #716

Merged

mkanoor closed this as completed in #716 Sep 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run_workflow_template only passes variables when prompt on launch selected in Workflow Job Template #622

run_workflow_template only passes variables when prompt on launch selected in Workflow Job Template #622

adworjan commented Nov 27, 2023

mkanoor commented Nov 27, 2023

adworjan commented Nov 27, 2023

mkanoor commented Nov 27, 2023

adworjan commented Nov 27, 2023

mkanoor commented Nov 27, 2023

adworjan commented Nov 27, 2023

mkanoor commented Nov 27, 2023

goldyfruit commented Dec 11, 2023

mkanoor commented Dec 12, 2023

goldyfruit commented Dec 12, 2023

run_workflow_template only passes variables when prompt on launch selected in Workflow Job Template #622

run_workflow_template only passes variables when prompt on launch selected in Workflow Job Template #622

Comments

adworjan commented Nov 27, 2023

Please confirm the following

Bug Summary

Environment

Steps to reproduce

Actual results

Expected results

Additional information

mkanoor commented Nov 27, 2023

adworjan commented Nov 27, 2023

mkanoor commented Nov 27, 2023

adworjan commented Nov 27, 2023

mkanoor commented Nov 27, 2023

adworjan commented Nov 27, 2023

mkanoor commented Nov 27, 2023

goldyfruit commented Dec 11, 2023

mkanoor commented Dec 12, 2023

goldyfruit commented Dec 12, 2023