FEATURE: Hand message over via cache

[daniellienert]

If you hand over some data (eg elasticsearch queue indexer with 500 nodes per job) to an isolated command run you exceed the escapeshellarg max input length. To avoid this, this change passes the payload via a separate cache.

Fixes: #25

[kdambekalns]

Nice one! Looks good by reading…

[hlubek]

I'd suggest to add persistent: true here, because this should not be flushed / whatever. It's a rare case that could happen during a deployment or flush, but anyway...

[hlubek]

Looks good, just added a note about using a persistent cache to prevent flush race conditions.

[daniellienert]

Thanks @hlubek - I guess the cases are really rare - but persistent does no harm.

[kdambekalns]

Yeah!

[bwaidelich]

I'm 4 years late to the party, but I'm afraid that this change introduced a weak spot into the architecture.
It was already hard to debug the job queue but this added complexity and a potential source of failure.

No hard feelings (obviously) but I wonder how we could solve the core issue otherwise.
Maybe we could limit the max size of a job somehow.
Putting 500 nodes into a single job defeats the purpose of a job queue IMO.

We should put this behind a feature flag at least I would say

[hlubek]

If it's about the isolated scenario: why not use STDIN to pass the data?
I agree the cache part introduces redundancy in data storage for the job queue - which will probably break somewhere.

[bwaidelich]

why not use STDIN to pass the data?

Great idea!
Not sure if we can make that work with Scripts::executeCommand() but using something more sophisticated like react/child-process might make sense anyways in order to have more control over the execution

[kitsunet]

Yep or https://symfony.com/doc/current/components/process.html