Understanding causes of windmill overhead

[some-user123]

I just migrated a typescript script to a flow in windmill. Windmill takes significantly longer to perform the same result:
Before: ~60s
In windmill: ~660s

Obivously, I get a lot of benefits from running it in windmill, but a factor of ~11 is significant. I would like to understand, where the overhead is coming from and how to possibly mitigate it.

What does this flow do?
It connects to a database, selects 1000 rows. Loops over the rows, for each row, it executes two deno scripts (which might make external API calls).
(But we can assume that the time spend waiting for external API calls isn't more than 60s in total. The script works also strictly sequential and won't do any API calls in parallel.)

To measure the overhead created by windmill, I created a dummy flow that just generates an array with 1000 entries, loops over them and executes two deno scripts that just print "Hello World". That dummy flow took 155s. So, 155s for all the magic windmill does (like passing input/output, validating input, storing all that, logging all output, etc.).

Adding the imports to the deno scripts, time of dummy flow increases to 190s. Makes sense, as the imports are executed 1000 times instead of once.

Where does the rest of the overhead come from (total overhead: 600s, explained overhead: 190s)? And how could it be reduced? I explicitely want to loop in windmill and not in deno to be able to see the individual runs of the "inner" scripts.

[rubenfiszel]

Thank you for the question and the investigation. I agree the overhead is significant.

I think you almost have the full answer in your investigation, it must come from what is non-dummy. Would you be able to put timestamp/measure time of execution of the non-dummy logic. Maybe the result will surprise you.

It seems to me that you have measured all the overhead that I can think of.
I would naively think that because of the api calls, you are IO bound and not cpu bound. However, for exhaustiveness, there is one "overhead" that could in theory have an effect is that v8 is an extremely powerful and magic runtime that use JIT optimization. It means that running a script 1000 times and running 1000 iterations inside a script is not the same because v8 can optimize how to execute the loop and spend time doing JIT for the inner loop logic. Again, I really do not think this is the case since you are doing API call but wanted to mention it.

[some-user123]

Good point, many thanks!

I measured the time on function/deno script, that performes the API call (depending on the data, we do ~150 API calls for the 1000 rows).

Non-windmill:

• Without API call: ~0.3ms
• With API call: ~100ms

Windmill:

• Without API call: ~20ms
• With API call: ~200ms

That's indeed a huge factor caused by v8's optimization magic.

Guess, that's the price you have to pay :-/

[rubenfiszel]

are you able to reproduce those timings doing a single deno run with a single api call ?

[rubenfiszel]

also if it's sufficient, you might be able to use the "REST" language which is essentially a limited deno runtime that is more lightweight to run

[some-user123]

Unfortunately, the script does a bit more than one can do without external libraries. I could split it up in multiple scripts, but I doubt, that would be faster.

Not sure, what you refer to ("reproduce those timings"). The pure console.time(); await fetch(...); console.timeEnd() prints the same time in windmill as with deno run. Above I measured the whole script around the fetch...

[gcleaves]

Along the lines of overhead, I find Windmill loops very, very slow. For example, a single Windmill Bun script which parses an array of 30 items takes a total of 1s on my small server. Using a Windmill flow and loop to parse the 30 items (in parallel) takes 20s. 20x slowness. (Running the script natively in Bun on my laptop takes .5s . )

As the OP states, some overhead is expected and acceptable, but 20s to loop through 30 items is surprising.

I'm using the community edition self hosted on Docker.

EDIT: My server was overloaded. I stopped some processes and the Windmill loop now takes 5.4s.

[fatherofinvention]

Hey @rubenfiszel - My PowerShell script which just prints the current date takes 8.167s to run in Windmill. Outside of Windmill inside an already running PowerShell process, it takes 1ms unsurprisingly. If I call it from bash: time pwsh -Command "Get-Date" it takes 1.8 seconds. So about a factor of 4.5x to run inside Windmill. I know you've been discussing Typescript here, but is there anyway to optimize performance for PowerShell? Any chance of somehow creating a continuously running worker in the self-hosted edition that executes commands if they are tagged as powershell ?

[rubenfiszel]

@fatherofinvention the overhead come mostly from windmill needing to be able to install powershell modules/library on the fly. The code is pretty naive and has the same performance than doing the sequence of commands that we do. I agree performances are atrocious and there are lots of room to grow. We would probably need more powershell expertise or to allow not installing library on the fly to improve performance.