Fix async i/o error and move startup script

[joyrex2001]

This PR starts the redpanda server with --smp 1 which resolves a Could not setup Async I/O: Resource temporarily unavailable. error when running tests using dind in k8s. This PR also moves the startup script to the /var/lib/redpanda/ folder which is owned by the redpanda user that runs the container.

See also: #19445

[cescoffier]

Thanks! The PR looks good to me. Let's see what the CI has to say!

[gastaldi]

Thanks! Is this PR supposed to resolve #19445 too completely?

[cescoffier]

@gastaldi yes, but we need to change the title of the issue.

The PR fixes potential startup issues in some environments.

[cescoffier]

@gastaldi just updated the issue title.

[Ladicek]

Hi @joyrex2001, this change (specifically the addition of --smp 1) breaks Kafka Dev Services for me locally. The Redpanda container fails to start with Could not initialize seastar: std::__1::system_error (error system:12, Cannot allocate memory).

Interestingly, if I add --memory 2G, it starts working again, but I don't know enough about Redpanda or Seastar to comment on why. I suspect that there's some automatic memory sizing logic that fails, but I don't really know.

I'm on Linux and have 32 GB of RAM. The container sees all those 32 GB of RAM (because we don't apply any memory cgroup limits), which is probably where the memory sizing logic gets confused.

Would that change (adding --memory 2G) be OK for you?

[joyrex2001]

According to the cli documentation not specifying a memory limit would actually imply that it will use all available memory, which makes me also suspect the automatic sizing logic fails.

I tried various combinations of the cli arguments they suggest in the docker quick start and also played with the --thread-affinity and --num-io-queues arguments. In our case the only thing that works in our docker-in-docker setup is adding the --smp 1, the others don’t have any functional impact at our end.

The redpanda docker quick start sets it to 1G, and also sets the --overprovisioned and --reserve-memory 0M arguments. Setting these here as well is probably the best way forward. Do these settings (specifically 1G) work at your end as well?

[Ladicek]

So in addition to --smp 1, also add --memory 1G --overprovisioned --reserve-memory 0M? I tried that and that works for me. And I agree we should probably copy what the Redpanda quick start documentation shows.

[joyrex2001]

Indeed :-) As this PR is already closed/merged, do you want me to create a new PR with the changes, or will you?

[Ladicek]

If you can submit a PR, that would be great, otherwise I'll do it tomorrow or the day after that. Thanks!