[RFC] Run external processes when starting web-server

[pierredup]

When embedding an app and building a static binary, there are some processes that I would like to run automatically when the php-server starts.
For example, if I want to distribute and app and someone runs the app, I would like to automatically run DB migrations, run the messenger:consume command, generate JWT tokens etc.

So I'm thinking of implementing something similar to how the Symfony CLI works (https://symfony.com/doc/current/setup/symfony_server.html#configuring-workers):

• Create a frankenphp.(yaml|json) config file which will be at the root of the embedded app
  • This config file can later be used to specify other default options (E.G path to the worker script)
• The config file can then specify commands that should be run when the php-server starts
  • E.G

    commands:
       - @php-cli bin/console lexik:jwt:generate-keypair 
       - @php-cli bin/console doctrine:migrations:migrate
       - @php-cli bin/console messenger:consume async
    
    # @php-cli is a magic value that means the same as `./binary php-cli`

When the php-server starts, run the Caddy server, along with each command in a separate process.

This will allow to distribute a binary for an application that "just-works" when it is run, without any additional set up or instructions required. It will also make upgrading an application easier, since something like the migrations can just be executed when the new binary is run. It will also help to add async processing to an application, without any additional setup or configuration.

I have an almost-working POC that I would be happy to create a PR for, but just want to get some general feedback and ideas first.

[withinboredom]

Personally, I'm not sure how I feel about this. On the one hand, it's a pretty interesting idea. On the other hand, we'd have to implement and maintain @php or @composer or any other tooling (like phive, or whatever). I'd prefer for us to implement a bin/frankenphp_startup.php, which includes some good environment variables and is called within a dedicated thread. Then, the community can implement whatever they want via composer plugins, custom scripts, YAML files, or whatever.

Personally, I would implement two scripts:

• bin/frankenphp_startup.php: called before the main application is initialized and the main application is not started until the script is complete. Run in a dedicated thread so workers don't get "poisoned" by whatever happens in the script. Any output will be appended as environment variables to the workers. For example, outputting KEYPAIR_LOCATION=/path/to/keys.json would allow you to read those environment variables in your script. Dynamic configuration could happen here, along with migrations, or possibly even user interaction.
• bin/frankenphp_sidecar.php: Runs once for each worker (or based on max_threads in cgi-mode) and is expected to never die. These "sidecar scripts" can consume queues, receive messages from requests (perhaps through a frankenphp_send_work(array $work) on the request side and a frankenphp_receive_work(int $maxWorkCount): array on the sidecar side) to send emails, etc.

That's some pretty epic scope creep from your initial proposal, but I feel like it would be much more flexible. Through people creating libraries, it could literally do anything.

[pierredup]

On the other hand, we'd have to implement and maintain @php or @composer or any other tooling (like phive, or whatever)

The idea is not to implement support for any tooling (except the built-in php-cli command from Frankenphp), but rather just have a way to run external commands when the application starts up.

bin/frankenphp_startup.php: called before the main application is initialized and the main application is not started until the script is complete.

This sounds like an interesting idea, but I think is a different feature than what is proposed here (I can open a new issue to discuss that separately if you want, since I think it's not completely related to the idea here).

bin/frankenphp_sidecar.php: Runs once for each worker (or based on max_threads in cgi-mode) and is expected to never die

The one issue here is that it will be a bit difficult to run multiple processes from a single file without a lot of additional work (E.G manually starting multiple processes and monitoring them using symfony/process). Some processes should also only run in a single process and not once for each worker (E.G a cron scrip or a scheduled command using symfony/scheduler)

Through people creating libraries, it could literally do anything.

I'm not sure what such a library would look like. This sounds a bit more restrictive since you would then either need to depend on other libraries vs just specifying a command to run when the application starts up (silly example: I can ship a Redis binary along with my application and have - ./bin/redis specified as a command. So when my application runs I can automatically have access to a running Redis instance without the user needing to install or configure anything)

[withinboredom]

The idea is not to implement support for any tooling (except the built-in php-cli command from Frankenphp), but rather just have a way to run external commands when the application starts up.

It'd be good to clarify that in the original post. However, if we are only going to support running a cli command, why do we need the @php at all? Having the @php adds additional complexity and error modes:

  - bin/some/command @php do_stuff

What should happen there, and how is it different from running a 'well-known' script that does whatever the user desires? What if the previous command fails and I don't care if it fails (or I do care if it fails)? What happens if a startup script never returns? Should we wait for it to return before starting the application? What happens if one of those commands fail, should it be restarted? What if it goes into an infinite loop of restarting?

I don't know, but I don't think frankenphp should be a process manager.

This sounds like an interesting idea, but I think is a different feature than what is proposed here

It sounds like what you are proposing, minus the additional indirection via a YAML file and process management. Though I am also not a fan of my proposal, as I don't want to use frankenphp as a queue.

The one issue here is that it will be a bit difficult to run multiple processes from a single file without a lot of additional work (E.G manually starting multiple processes and monitoring them using symfony/process). Some processes should also only run in a single process and not once for each worker (E.G a cron scrip or a scheduled command using symfony/scheduler)

There are data structures (semaphores, mutexes, etc) to provide synchronization, and those could be used to coordinate things between threads. It also allows things to fall back on other running threads nearly instantly in the event of a crash.

So when my application runs I can automatically have access to a running Redis instance without the user needing to install or configure anything

Please don't do this. Things like this are how Mongo databases ended up being exposed to the internet with no authentication, leaking user data everywhere, or how I have 15 copies of a database engine running on my computer. Just use SQLite or similar embedded databases and allow configuration for other engines. Javascript has already gone down that road and it didn't end well.

[pierredup]

So when my application runs I can automatically have access to a running Redis instance without the user needing to install or configure anything

Please don't do this.

Bad example, I know. I was just trying to think of ideas of running external scripts that's not tied to a PHP script inside the embedded app to try and explain the idea

[dunglas]

Maybe we could leverage existing Caddy plugins to do that, and create new ones if needed? It would be better to have generic solutions at the Caddy level instead of FrankenPHP-specific code (it will benefit a larger community).

There is https://github.com/abiosoft/caddy-exec for one-off commands and https://github.com/Baldinof/caddy-supervisor for background tasks.

[pierredup]

There is abiosoft/caddy-exec for one-off commands and Baldinof/caddy-supervisor for background tasks.

This looks like it will cover my use-case perfectly, thanks!

[bpolaszek]

Agreed with @dunglas - Baldinof/caddy-supervisor is a perfect fit for this (Messenger workers), I'd just like to figure out how to feed frankenphp with a custom Caddyfile 😅