Documentation feedback

[bithavoc]

I want to track here some of the feedback I already shared with the team:

1. Single Node Production Setup states XFS as a Requirement, turns out you can totally use redpanda without XFS, this was a show-stopper for me where I'm not looking for the best performance, I just don't want to deal with ZK so a Single node without XFS should work, my recommendation would be to either create a "tier" before "Single Node Production Setup" or the same setup but with two sections: Minimal Requirements(no XFS) and Optimal Requirements(XFS) with a link to the deployment automation. Reference: https://news.ycombinator.com/item?id=25110390

2. A compatibility table with Kafka would be great, I was specifically interested in topic compaction and retention and couldn't find any mention or info around it [0] https://news.ycombinator.com/item?id=25113978

3. Apparently packagecloud is being phased out but there are still some references in the documentation, reference https://vectorized.io/documentation (spotted by Alex, I had no clue)

[emaxerrno]

cc: @rkruze @yougotashovel

[0x5d]

Thanks for the feedback @bithavoc! We'll get on this asap.

[emaxerrno]

@0x5d i think @dotnwat uses area/<thing> and has a color coding scheme. might be worth checking w/ him.

[0x5d]

Oh, that label was already there so I woke it up from its slumber. I'll create a new one.

[bithavoc]

Also, I think the same place where the docs invite the visitors to run one of those package one-liners should also mention RAM requirements, when I tried the DEB script in Ubuntu 20 LTS in a node with 2GB of RAM I get:

Job for redpanda.service failed because the control process exited with error code.
See "systemctl status redpanda.service" and "journalctl -xe" for details.

logs say:

Dec 01 00:14:32 test-red-panda rpk[2683]: INFO  2020-12-01 00:14:32,774 [shard 0] syschecks - Writing pid file "/var/lib/redpanda/data/pid.lock"
Dec 01 00:14:32 test-red-panda rpk[2683]: ERROR 2020-12-01 00:14:32,774 [shard 0] syschecks - Memory: '499122176' below recommended: '1073741824'
Dec 01 00:14:32 test-red-panda rpk[2683]: INFO  2020-12-01 00:14:32,775 [shard 0] redpanda::main - application.cc:88 - Failure during startup: std::runtime_error (Memory: '499122176' below recommended: '1073741824')
Dec 01 00:14:32 test-red-panda systemd[1]: redpanda.service: Main process exited, code=exited, status=1/FAILURE
Dec 01 00:14:32 test-red-panda systemd[1]: redpanda.service: Failed with result 'exit-code'.
Dec 01 00:14:32 test-red-panda systemd[1]: Failed to start Redpanda, the fastest queue in the West..

So there's a minimum of memory but no mention anywhere in the docs, I'm sorry if I missed it if it's there, the only requirements I could see in the instructions page was something about XFS and port 9092, nothing else.

You may be wondering why I'm trying redpanda in a VM with 2GB of RAM, well, it's my perception that given this project was built in C++ and not Java, then the minimal requirements would be more forgiving, I remember running Kafka in Docker in a VM with similar specs, it was painful due the non root user requirements in the bitnami kafka images but is totally doable.

[emaxerrno]

i think that's a good point. @0x5d - should we default to 1core ?

[0x5d]

I'm all about making the first impression as seamless as possible. I think @rkruze had also mentioned that the dev mode should be the default. If we make that the case, then redpanda.developer_mode would be true and the minimum memory check wouldn't be strict. I think that's the way to go.

[rkruze]

Fully agree that redpanda.developer_mode should be true.

[bithavoc]

Sorry I keep hijacking this thread with my experience, I know this issue was originally about about documentation but maybe this is somewhat related.

First, I have Redpanda running 🚀 yay!, I was able to use rpk to produce and consume dummy messages, but even when my app is ready to use any kafka cluster I still can't use Redpanda.

Here's my situation: given I have no knowledge of how to operate a redpanda cluster and there is no docker image, I'm being forced to use a dedicated server to act as a single-node redpanda cluster, I'm running it in a VM with VM-level backups so I can restore the entire thing without reconstructing my cluster. Also, my kafka app is in a kubernetes cluster in a different cloud, no VPC peering available nor I would to do it if available, so I need my app to connect to my redpanda instance via public traffic but the default setup doesn't include and tls configuration for kafka_api, hence my suggestion: the DEV package should at least automatically generate a self-signed TLS and configure it for me so I can connect securely from anywhere just by opening the right ports in the firewall.

A rpk cert command in a similar spirit to cockroach's cert command to create client certs would be even better but not ideal, I think cockroach's approach is a bit too much due the hassle of creating and maintaining client certificates when deploying apps and clusters from scratch for testing and reproducible environments, etc

IMO both developer and production mode, at minimum, should work as follows:

• kafka_api_tls.require_client_auth: false
• SASL/PLAIN? ( I just want to use User and Password with a self-signed CA, no client verification required)

This is where the documentation could shine:

• Authentication Mechanisms supported by Redpanda
• How to properly setup TLS, it's automatic by default, but just in case the user wants to do it or rotate secrets, then describe how to generate files for kafka_api_tls.* with openssl, cfssl, etc

btw, in this page it seems like kafka_api_tls is duplicated.

[dotnwat]

hey @bithavoc we are working on acls right now and part of that is to tackle authentication. today we only have ssl + plain, but by the end of the december we expect to have sasl+scram in addition to plain, so you could have a common SSL+SCRAM (user and password) setup.

we'll keep working on the docs, and please don't hesitate to dump more feedback here, it's great!

[bmansheim]

@dotnwat It seems like some of the issues here have been addressed. Can you review it and pick out remaining action items?

btw, in this page it seems like kafka_api_tls is duplicated.

This was resolved.

[dotnwat]

@bmansheim i think we can close this now. I split out the idea for the compatibility table into a separate ticket to keep things simpler. everything else seemed ok.

[sanjaynv]

Also, I think the same place where the docs invite the visitors to run one of those package one-liners should also mention RAM requirements, when I tried the DEB script in Ubuntu 20 LTS in a node with 2GB of RAM I get:

Job for redpanda.service failed because the control process exited with error code.
See "systemctl status redpanda.service" and "journalctl -xe" for details.

logs say:

Dec 01 00:14:32 test-red-panda rpk[2683]: INFO  2020-12-01 00:14:32,774 [shard 0] syschecks - Writing pid file "/var/lib/redpanda/data/pid.lock"
Dec 01 00:14:32 test-red-panda rpk[2683]: ERROR 2020-12-01 00:14:32,774 [shard 0] syschecks - Memory: '499122176' below recommended: '1073741824'
Dec 01 00:14:32 test-red-panda rpk[2683]: INFO  2020-12-01 00:14:32,775 [shard 0] redpanda::main - application.cc:88 - Failure during startup: std::runtime_error (Memory: '499122176' below recommended: '1073741824')
Dec 01 00:14:32 test-red-panda systemd[1]: redpanda.service: Main process exited, code=exited, status=1/FAILURE
Dec 01 00:14:32 test-red-panda systemd[1]: redpanda.service: Failed with result 'exit-code'.
Dec 01 00:14:32 test-red-panda systemd[1]: Failed to start Redpanda, the fastest queue in the West..

So there's a minimum of memory but no mention anywhere in the docs, I'm sorry if I missed it if it's there, the only requirements I could see in the instructions page was something about XFS and port 9092, nothing else.

You may be wondering why I'm trying redpanda in a VM with 2GB of RAM, well, it's my perception that given this project was built in C++ and not Java, then the minimal requirements would be more forgiving, I remember running Kafka in Docker in a VM with similar specs, it was painful due the non root user requirements in the bitnami kafka images but is totally doable.

How do you resolve the above problem...? Because i am beginner to this and facing the same issues in redpanda ..

[emaxerrno]

@sanjaynv - it's mentioned in the thread. see developer_mode:true