Creating/Updating too many job templates in parallel seems to be too much for controller API

[dbk-rabel]

Hi.

We are using the controller_configuration to configure our controller.
Here is a snippet from our playbook:

- name: Ensure organizations, teams, applications and credentials are present
  ansible.builtin.include_role:
    name: infra.controller_configuration.dispatch
  vars:
    controller_configuration_dispatcher_roles:
      - {role: organizations, var: controller_organizations, tags: organizations}
      - {role: teams, var: controller_teams, tags: teams}
      - {role: users, var: controller_user_accounts, tags: users}
      - {role: applications, var: controller_applications, tags: applications}
      - {role: credentials, var: controller_credentials, tags: credentials}
      - {role: credential_types, var: controller_credential_types, tags: credential_types}

- name: Ensure organizations have their pah credentials attached
  when: infrastructure_privileged_mode
  ansible.builtin.include_role:
    # We cannot use the the dispatcher role here, because it overrides 'assign_galaxy_credentials_to_org'
    name: infra.controller_configuration.organizations
  vars:
    assign_galaxy_credentials_to_org: true

- name: Ensure projects, inventories, inventory sources and execution environments are present
  ansible.builtin.include_role:
    name: infra.controller_configuration.dispatch
  vars:
    controller_configuration_dispatcher_roles:
      - {role: projects, var: controller_projects, tags: projects}
      - {role: inventories, var: controller_inventories, tags: inventories}
      - {role: inventory_sources, var: controller_inventory_sources, tags: inventory_sources}
      - {role: execution_environments, var: controller_execution_environments, tags: execution_environments}

- name: Ensure inventory sources have tried to sync
  ansible.builtin.include_role:
    name: infra.controller_configuration.inventory_source_update
    apply:
      ignore_errors: true

- name: Ensure job templates, workflow job templates and roles are present
  ansible.builtin.include_role:
    name: infra.controller_configuration.dispatch
  vars:
    controller_configuration_dispatcher_roles:
      - {role: job_templates, var: controller_templates, tags: job_templates}
      - {role: workflow_job_templates, var: controller_workflows, tags: workflow_job_templates}
      - {role: roles, var: controller_roles, tags: roles}
      - {role: schedules, var: controller_schedules, tags: schedules}

- name: Ensure that the objects which are not defined in git are absent
  when: infrastructure_cleanup
  block:
    - name: Generate a list which contains all objects which should be present on the controller or deleted from the controller
      ansible.builtin.include_tasks:
        file: object_diff.yml

    - name: Ensure workflow job templates, job templates, inventory sources, inventories and projects which are not defined in git are absent
      ansible.builtin.include_role:
        name: infra.controller_configuration.dispatch
      vars:
        controller_configuration_dispatcher_roles:
          - {role: workflow_job_templates, var: controller_workflows, tags: workflow_job_templates}
          - {role: job_templates, var: controller_templates, tags: job_templates}
          - {role: inventory_sources, var: controller_inventory_sources, tags: inventory_sources}
          - {role: inventories, var: controller_inventories, tags: inventories}
          - {role: projects, var: controller_projects, tags: projects}

    - name: Ensure credentials and organizations which are not defined in git are absent
      ansible.builtin.include_role:
        name: infra.controller_configuration.dispatch
      vars:
        controller_configuration_dispatcher_roles:
          - {role: credentials, var: controller_credentials, tags: credentials}
          - {role: organizations, var: controller_organizations, tags: organizations}

This more or less works fine.

But now we seem to be hit by Errno 104 "Connection reset by peer" at async_status for job templates. (Interestingly only at the bottom after the object_diff.)

It seems to have something to do with the controller API not being able to handle so many calls at the same time. If we just delete a few job templates from controller_job_templates everything is working fine again.

It seems there is no way to slow things down a little bit, is there? Because that is what we would need.

We tried increasing controller_configuration_job_templates_async_delay , but it does not seem to work. The problem is already on the first try.

Yours
David

[djdanielsson]

yes this is likely an issue with object_diff it was quickly written based off an idea and really needs to be rewritten from the ground up to fix a lot of issues with it.

[dbk-rabel]

Thanks for the reply.

But I do exactly understand what is happening.

Having too many API calls in a too short amount of time is probably the issue. But I don't see a way to slow down the execution of the controller_configurationt roles. Or is there any?

Yours
David