Increase max reconciliation time

[alexec]

Summary

We'd want to support 10k+ node workflows, and these tend to fail to schedule all their nodes within in the 10s deadline. In fact, it is possible for a workflow to get so large it can never fully schedule its nodes.

In #2705 (May) we increased the default number of workflow-workflows from 8 to 32.

Let`s increase the reconciliation time (or make it configurable)

Use Cases

10k+ node workflows.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

I've done some experiments and I can't see the benefit on 250 node workflows. I strongly expect benefits for larger numbers.

[akloss-cibo]

I pulled the workflow nodes out of the postgresql for one of our workflows that gets stuck in Running while making no progress. Postgresql takes a while just to process the row in a trivial explain:

postgres=# select count(*) from argo_workflows;
 count
-------
     2
(1 row)

postgres=# explain analyze select json_each(nodes) as node from argo_workflows where uid = 'b66dd218-9555-4745-8cb4-2e38fd014762';
                                                  QUERY PLAN
---------------------------------------------------------------------------------------------------------------
 ProjectSet  (cost=0.00..2.04 rows=200 width=32) (actual time=1788.320..1929.525 rows=187695 loops=1)
   ->  Seq Scan on argo_workflows  (cost=0.00..1.02 rows=2 width=18) (actual time=0.024..0.028 rows=1 loops=1)
         Filter: ((uid)::text = 'b66dd218-9555-4745-8cb4-2e38fd014762'::text)
         Rows Removed by Filter: 1
 Planning Time: 0.070 ms
 Execution Time: 1936.627 ms
(6 rows)

postgres=#

I did pull down the nodes and:

% time jq . ~/tmp/parsed.json > /dev/null
jq . ~/tmp/parsed.json > /dev/null  19.24s user 0.76s system 97% cpu 20.409 total
% du -h parsed.json
224M	parsed.json
%

[alexec]

You should not be querying from argo_workflows using uid. There is no index, so you'll end up sequentially scanning the table.

primary key(clustername,uid,version)

In fact, you should never be querying from that table. It is internal to Argo Workflows.

Can you try argoproj/workflow-controller:latest please?

[akloss-cibo]

Since there are only two rows in the table, I think the overhead from the table scan is nominal compared to reading the json.

Yep, we'll give it a go.

Based on another issue, I also set ALL_POD_CHANGES_SIGNIFICANT=true in the workflow-controller's environment.

[akloss-cibo]

We did have some success with the :latest image in that our workflow made it to Failed status. When trying to retry through the web UI, there's an error. The log says

time="2020-11-04T15:33:10Z" level=error msg="finished unary call with code Unknown" error="upper: no more rows in this result set" grpc.code=Unknown grpc.method=RetryWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2020-11-04T15:33:10Z" grpc.time_ms=25.877 span.kind=server system=grpc

Any ideas?

[alexec]

huh - that error is related to offloading - have you turned it on and the off?

[akloss-cibo]

AFAIK, we have not. Are you suggesting that we try that?

[alexec]

No, you should never see that error - it indicates that the data was deleted from the database, but not from etcd. I don't know how this can happen and could be a separate unrelated bug.

Have you ever seen this before?

[akloss-cibo]

I think that was the first time we've gotten a workflow to finish without lots of manual intervention. We did run it again overnight and have the same behavior, by which I mean the workflow runs, fails, and then cannot be retried. The tables I think are relevant are empty:

postgres=# select count(*) from argo_workflows;
 count
-------
     0
(1 row)

postgres=# select count(*) from argo_archived_workflows;
 count
-------
     0
(1 row)

postgres=#

[akloss-cibo]

We've given another large workflow a try and it has ended up stuck Running with no pods for the workflow. At this point, we have

    - name: OFFLOAD_NODE_STATUS_TTL
      value: 4320m
    - name: ALL_POD_CHANGES_SIGNIFICANT
      value: "true"

set for the workflow controller. We periodically see

2020-11-09 19:18:54.182 UTC [8383] FATAL:  connection to client lost
2020-11-09 19:18:54.182 UTC [8383] STATEMENT:  SELECT
	        "uid", "version", "nodes"
	        FROM "argo_workflows"
	      WHERE ("clustername" = $1 AND "namespace" = $2)

(empty lines removed) in the postgresql server's stdout. I think this may be coming from argo-server, not the workflow-controller. We went up to a pretty big pod for argo-server (8 CPU,s 32 GI memory) to no avail; attempting to use the UI times out and occasionally causes the argo-server pod to either OOM (when it was configured a little smaller) or just fail its health check and restart.

Are there other things we could be trying?

[alexec]

Could we have a Zoom?

[akloss-cibo]

Sure... I just joined the argo slack; is that a good way to get linked up?

[alexec]

Ping me on Slack.

[akloss-cibo]

We've switched to a workflow-of-workflows for this particular task. I've spent a little time trying to create a simpler way to reproduce issues without much luck, but we'll reach out if we find something that we're stuck on.