add sequential execution mode option #103

yu-fuku · 2023-03-28T13:06:54Z

discussion stepci/stepci#122
stepci stepci/stepci#123

mishushakov · 2023-03-28T13:36:09Z

I believe some users would prefer more control over the concurrency (eg. 1 to Infinity)
Also this should be a workflow-level setting, with an option to overwrite from CLI

Have a look at p-map by sindresorhus:
https://github.com/sindresorhus/p-map

Maybe refactor the run function with it accordingly?

mishushakov · 2023-03-28T13:37:35Z

Btw. sindre has moved all his packages to ESM modules, which we don't support. If you see a related error, just install an older version of p-map

yu-fuku · 2023-03-29T11:15:30Z

Thanks for checking and answering.

I believe some users would prefer more control over the concurrency (eg. 1 to Infinity)

I'd prefer to have user control, as you say.

Let me ask you a few questions

Also this should be a workflow-level setting,

Am I correct in understanding that the file specified in the include syntax is the target?
(I think only one workflow will work unless you specify multiple files in the include)

For example, for the following workflow files, allow the user to choose which of the included files to execute sequentially or in concurrency?

version: "1.1"
name: Status Check
include:
  - sequentially_1.yml
  - sequentially_2.yml
  - concurrency.yml

sequentially_1.yml and sequentially_2.yml are executed sequentially (sequentially_2.yml is executed when sequentially_1yml is completed)
concurrency.yml is parallel execution
Allow users to select files to run sequentially on a file-by-file basis?

with an option to overwrite from CLI

What value do you expect to input from CLI as option?
(Name of files you want to run sequentially(or concurrency)?)
I'm sorry, I can't imagine much about the content of this answer.

mishushakov · 2023-03-29T11:23:02Z

Workflow level:

version: "1.1"
name: Status Check
config:
  concurrency: 3 # Run 3 tests concurrently
include:
  - 1.yml
  - 2.yml
  - 3.yml

CLI option:

stepci run workflow.yml --concurrency=3

Hope this helps. Let me know if you need more

mishushakov · 2023-03-29T11:23:51Z

When config.concurency is not specified it should default to Infinite

mishushakov · 2023-03-29T11:27:01Z

There's also this library, which is similar to p-map:
https://www.npmjs.com/package/p-limit

yu-fuku · 2023-03-30T04:01:10Z

Thanks for replay.
Your answers are so helpful.

You mean to allow parallel execution of included files with specified number of parallel.

Please give me a few days to fix the current implementation and update the pull request.

mishushakov · 2023-03-30T10:03:12Z

Of course! Glad to help further.

Yes, if you set concurrency setting is set to 0, the tests should run sequentially, just as you wanted

Don't feel pressured though. In open-source you can contribute at your own pace 😎

mishushakov · 2023-04-12T14:48:01Z

Hi, @yu-fuku
Let me know if you're stuck and need any help 😁

yu-fuku · 2023-04-15T08:55:55Z

Sorry it took so long to respond.

The number of parallel executions can now be controlled using p-limt.
Please check and comment.

mishushakov · 2023-04-15T10:35:27Z

No worries! Glad you took your time to update the pull request.

Have you verified it works?

yu-fuku · 2023-04-18T04:37:39Z

Yes. I have confirmed the following operation.

1, No config.concurrency or cli arguments set → confirm that everything runs in parallel
2, Parallel execution with the number of executions set in config.concurrency
3, Parallel execution with the value set in the cli argument(highest priority)

mishushakov · 2023-04-18T06:43:19Z

If you set it to 0 does it run in a sequence?

yu-fuku · 2023-04-18T07:34:33Z

Sorry I didn't check when I put the 0 in.

p-limit did not seem to accept values less than 0
https://github.com/sindresorhus/p-limit/blob/v3.1.0/index.js#L5

So I modified runner so that it is executed sequentially when a value less than or equal to 0 is entered.
Please check and comment.

I confirmed that when I incorporate this modification and enter 0, sequential execution is being performed.

mishushakov

Pretty much there (thanks for updating the pull request). I have very small changes to suggest 😄

mishushakov · 2023-04-18T15:14:14Z

src/index.ts

-  const testResults = await Promise.all(Object.entries(tests).map(([id, test]) => runTest(id, test, schemaValidator, options, workflow.config, env, credentials)))
+  let concurrency_num = Object.keys(tests).length
+
+  if(workflow.config?.concurrency || workflow.config?.concurrency == 0) {


remove the workflow.config?.concurrency == 0 condition

mishushakov · 2023-04-18T15:14:22Z

src/index.ts

+  if(workflow.config?.concurrency || workflow.config?.concurrency == 0) {
+    concurrency_num = workflow.config?.concurrency
+  }
+  if(options?.concurrency || options?.concurrency == 0){


remove the options?.concurrency == 0 condition

mishushakov · 2023-04-18T15:15:09Z

src/index.ts

+    concurrency_num = 1
+  }
+
+  let limit = pLimit(concurrency_num)


let should be const

mishushakov · 2023-05-24T10:10:43Z

Hello, let me know if you have any updates?

fu-yuta · 2023-06-05T01:02:40Z

sorry I wasn't able to respond to your comment
I have responded now, so could you please check?

mishushakov · 2023-06-05T18:18:10Z

No worries, merging it!

yu-fuku mentioned this pull request Mar 28, 2023

add sequential execution mode option stepci/stepci#123

Merged

yu-fuku force-pushed the sequential_mode branch 2 times, most recently from 45f0ab6 to a354502 Compare March 30, 2023 04:53

yu-fuku force-pushed the sequential_mode branch from a354502 to e807312 Compare April 15, 2023 08:50

Enabled to specify the number of parallel executions

d188d52

yu-fuku force-pushed the sequential_mode branch from e807312 to d188d52 Compare April 15, 2023 08:51

Sequential execution if a value less than or equal to 0 is entered

4e949c4

mishushakov requested changes Apr 18, 2023

View reviewed changes

Minor fix

544feb7

mishushakov merged commit 0663956 into stepci:main Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add sequential execution mode option #103

add sequential execution mode option #103

yu-fuku commented Mar 28, 2023 •

edited

mishushakov commented Mar 28, 2023 •

edited

mishushakov commented Mar 28, 2023 •

edited

yu-fuku commented Mar 29, 2023 •

edited

mishushakov commented Mar 29, 2023

mishushakov commented Mar 29, 2023

mishushakov commented Mar 29, 2023

yu-fuku commented Mar 30, 2023

mishushakov commented Mar 30, 2023 •

edited

mishushakov commented Apr 12, 2023 •

edited

yu-fuku commented Apr 15, 2023

mishushakov commented Apr 15, 2023

yu-fuku commented Apr 18, 2023

mishushakov commented Apr 18, 2023 •

edited

yu-fuku commented Apr 18, 2023

mishushakov left a comment

mishushakov Apr 18, 2023

mishushakov Apr 18, 2023

mishushakov Apr 18, 2023

mishushakov commented May 24, 2023

fu-yuta commented Jun 5, 2023

mishushakov commented Jun 5, 2023

add sequential execution mode option #103

add sequential execution mode option #103

Conversation

yu-fuku commented Mar 28, 2023 • edited

mishushakov commented Mar 28, 2023 • edited

mishushakov commented Mar 28, 2023 • edited

yu-fuku commented Mar 29, 2023 • edited

mishushakov commented Mar 29, 2023

mishushakov commented Mar 29, 2023

mishushakov commented Mar 29, 2023

yu-fuku commented Mar 30, 2023

mishushakov commented Mar 30, 2023 • edited

mishushakov commented Apr 12, 2023 • edited

yu-fuku commented Apr 15, 2023

mishushakov commented Apr 15, 2023

yu-fuku commented Apr 18, 2023

mishushakov commented Apr 18, 2023 • edited

yu-fuku commented Apr 18, 2023

mishushakov left a comment

Choose a reason for hiding this comment

mishushakov Apr 18, 2023

Choose a reason for hiding this comment

mishushakov Apr 18, 2023

Choose a reason for hiding this comment

mishushakov Apr 18, 2023

Choose a reason for hiding this comment

mishushakov commented May 24, 2023

fu-yuta commented Jun 5, 2023

mishushakov commented Jun 5, 2023

yu-fuku commented Mar 28, 2023 •

edited

mishushakov commented Mar 28, 2023 •

edited

mishushakov commented Mar 28, 2023 •

edited

yu-fuku commented Mar 29, 2023 •

edited

mishushakov commented Mar 30, 2023 •

edited

mishushakov commented Apr 12, 2023 •

edited

mishushakov commented Apr 18, 2023 •

edited