Couchdb stops activity and then after some time we have always timeouts

[raulmartinezr]

Description

We use dockerized couchdb in local development environment to test applications.
The procedure for each test:

• create a database in couchdb
• test whatever we need to test inside this DB
• delete it in case it's not necessary anymore

We have aprox 70 tests

For each database a few documents are created

• less than 10 documents usually (max 50 when we test indexes created by ddoc)
• 2 design documents, one with two mango indexes and the other one with 1 mango index

If we execute tests module by module (each module contain between 1 and 10 tests ) there is not any issue, but if we execute all at the same time (sequentially, without parallelism), then at some point tests hangs

We observed

• Couchdb stops writting logs (CPU consumption decreases)
• We still have responses to quieries for some time, 1min aprox
• Then we have not responses from couchdb anymore

Example (It's not always the same)

• Last logs written (querying a view with /master/_partition/iam/_find ). After that CPU consumption of couchdb goes down.

[debug] 2020-08-17T19:30:05.663402Z nonode@nohost <0.20917.0> a13fe062bc no record of user admin
[debug] 2020-08-17T19:30:05.663457Z nonode@nohost <0.20917.0> a13fe062bc timeout 600
[debug] 2020-08-17T19:30:05.663495Z nonode@nohost <0.20917.0> a13fe062bc Successful cookie auth as: "admin"
[notice] 2020-08-17T19:30:05.665187Z nonode@nohost <0.20917.0> a13fe062bc 127.0.0.1:5984 172.28.0.1 admin POST /master/_partition/iam/_find 200 ok 2
[debug] 2020-08-17T19:30:05.677477Z nonode@nohost <0.20917.0> be296403e4 no record of user admin
[debug] 2020-08-17T19:30:05.677539Z nonode@nohost <0.20917.0> be296403e4 timeout 600
[debug] 2020-08-17T19:30:05.677570Z nonode@nohost <0.20917.0> be296403e4 Successful cookie auth as: "admin"

• Request captured with tcpdump
  

• After some time (1min 9s), timeouts
  

HW and environment

• Host machine runs Ubuntu 20.04, with 8cores and 16Gb RAM and 512GB SSD (250free)
• Docker container has not any limitation in CPU/Memory/Space
• Couchdb configured as single node (full configuration below)

This is how the container processes looks. Earlang (beam.smp) is the most consuming, with peaks of 70% CPU

Theads of beam.smp: Just when issue happens
I monitored threads during the whole process. Just before the crash, it seems scheduler threads increase the activity

top - 20:55:05 up  2:58,  0 users,  load average: 3.93, 2.31, 2.35
Threads:  46 total,   0 running,  46 sleeping,   0 stopped,   0 zombie
%Cpu(s): 46.2 us,  4.3 sy,  0.0 ni, 49.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15687.8 total,   1947.6 free,   7294.3 used,   6445.9 buff/cache
MiB Swap:    980.0 total,    980.0 free,      0.0 used.   7200.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     54 couchdb   20   0 4568544  62092  10904 S  40.0   0.4   0:07.25 2_scheduler
     53 couchdb   20   0 4568544  62092  10904 S  33.3   0.4   0:20.74 1_scheduler
     55 couchdb   20   0 4568544  62092  10904 S  33.3   0.4   0:06.63 3_scheduler
     56 couchdb   20   0 4568544  62092  10904 S  20.0   0.4   0:05.66 4_scheduler
     37 couchdb   20   0 4568544  62092  10904 S   6.7   0.4   0:00.03 async_2
      6 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.03 beam.smp
     34 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.00 sys_sig_dispatc
     35 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.00 sys_msg_dispatc
     36 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.18 async_1
     38 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.14 async_3
.....

Any idea about what could be the cause? Any hint would be appreaciated.

Steps to Reproduce

There is not a fixed trigger for the issue.

Expected Behaviour

We would expect couchdb can handle this load even with docker. It's not heavy, during tests we have 15operations per second max.

Your Environment

{"couchdb":"Welcome","version":"3.1.0","git_sha":"ff0feea20","uuid":"b17edbd1de7d1504022d6f359ff9a4f8","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

• CouchDB version used: 3.1.0
• Browser name and version: Not relevant
• Operating system and version: Couchdb docker@Ubuntu20.04

Additional Context

Full couchdb configuration

Configuration Settings:
  [admins] admin="******"
  [attachments] compressible_types="text/*, application/javascript, application/json, application/xml"
  [attachments] compression_level="8"
  [chttpd] backlog="512"
  [chttpd] bind_address="any"
  [chttpd] max_db_number_for_dbs_info_req="100"
  [chttpd] port="5984"
  [chttpd] prefer_minimal="Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary"
  [chttpd] require_valid_user="false"
  [chttpd] server_options="[{backlog, 512}, {acceptor_pool_size, 64}, {max, 4096}]"
  [chttpd] socket_options="[{sndbuf, 262144}, {nodelay, true}]"
  [cluster] n="3"
  [cluster] q="2"
  [cors] credentials="false"
  [couch_httpd_auth] allow_persistent_cookies="true"
  [couch_httpd_auth] auth_cache_size="50"
  [couch_httpd_auth] authentication_db="_users"
  [couch_httpd_auth] authentication_redirect="/_utils/session.html"
  [couch_httpd_auth] iterations="10"
  [couch_httpd_auth] require_valid_user="false"
  [couch_httpd_auth] secret="00464db7ba8beb6a5915e4f5dbd03a49"
  [couch_httpd_auth] timeout="600"
  [couch_peruser] database_prefix="userdb-"
  [couch_peruser] delete_dbs="false"
  [couch_peruser] enable="false"
  [couchdb] attachment_stream_buffer_size="4096"
  [couchdb] changes_doc_ids_optimization_threshold="100"
  [couchdb] database_dir="./data"
  [couchdb] default_engine="couch"
  [couchdb] default_security="admin_only"
  [couchdb] file_compression="snappy"
  [couchdb] max_dbs_open="10000"
  [couchdb] max_document_size="8000000"
  [couchdb] os_process_timeout="20000"
  [couchdb] single_node="true"
  [couchdb] users_db_security_editable="false"
  [couchdb] uuid="b17edbd1de7d1504022d6f359ff9a4f8"
  [couchdb] view_index_dir="./data"
  [couchdb_engines] couch="couch_bt_engine"
  [csp] enable="true"
  [fabric] request_timeout="infinity"
  [feature_flags] partitioned||*="true"
  [httpd] allow_jsonp="false"
  [httpd] authentication_handlers="{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}"
  [httpd] bind_address="127.0.0.1"
  [httpd] enable_cors="false"
  [httpd] enable_xframe_options="false"
  [httpd] max_http_request_size="4294967296"
  [httpd] port="5986"
  [httpd] secure_rewrites="true"
  [httpd] socket_options="[{sndbuf, 262144}]"
  [indexers] couch_mrview="true"
  [ioq] concurrency="10"
  [ioq] ratio="0.01"
  [ioq.bypass] compaction="false"
  [ioq.bypass] os_process="true"
  [ioq.bypass] read="true"
  [ioq.bypass] shard_sync="false"
  [ioq.bypass] view_update="true"
  [ioq.bypass] write="true"
  [log] level="debug"
  [log] writer="stderr"
  [query_server_config] os_process_limit="2000"
  [query_server_config] os_process_soft_limit="1000"
  [query_server_config] reduce_limit="true"
  [replicator] connection_timeout="30000"
  [replicator] http_connections="20"
  [replicator] interval="60000"
  [replicator] max_churn="20"
  [replicator] max_jobs="500"
  [replicator] retries_per_request="5"
  [replicator] socket_options="[{keepalive, true}, {nodelay, false}]"
  [replicator] ssl_certificate_max_depth="3"
  [replicator] startup_jitter="5000"
  [replicator] verify_ssl_certificates="false"
  [replicator] worker_batch_size="500"
  [replicator] worker_processes="4"
  [ssl] port="6984"
  [uuids] algorithm="sequential"
  [uuids] max_count="1000"
  [vendor] name="The Apache Software Foundation"

Some errors identified in the startup. They seem not relevant for the case.

• _users db does not exists (but seems to be created afterwards)

[error] 2020-08-17T19:18:47.731011Z nonode@nohost emulator -------- Error in process <0.372.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,374}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}

• I suppose without any effect as long as it's configured as single node [couchdb] single_node="true"

[error] 2020-08-17T19:18:47.799365Z nonode@nohost <0.457.0> -------- Request to create N=3 DB but only 1 node(s)
[error] 2020-08-17T19:18:47.812920Z nonode@nohost <0.457.0> -------- Request to create N=3 DB but only 1 node(s)

[willholley]

@raulmartinezr are you using Docker volumes to mount a CouchDB data volume or using a storage driver to write to the Docker filesystem directly? If the latter, which driver are you using?

[raulmartinezr]

Hi @willholley !

we are using tmfs

    tmpfs:
      - /opt/couchdb/data

As long as we do not need to persist data between runs, that option was usefull.

BR,
Raul

[willholley]

Thanks @raulmartinezr. Just out of interest, are you able to reproduce the problem using podman instead of Docker? Afaik podman will use the same container runtime (runc) as Docker so it would help narrow down the problem to something Docker-specific vs a container/CouchDB issue.

[raulmartinezr]

Hi @willholley ,

I tried to reproduce with podman but it was not possible. With podman it's working fine. Seems to be something related to docker daemon.

What do you think?

Thanks,
Raul

[willholley]

Thanks @raulmartinezr - that's certainly interesting. What version of Docker are you running (output of docker version would be helpful)?

[raulmartinezr]

Hello!

Thats what I have

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:45:44 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:44:15 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[willholley]

@raulmartinezr that looks recent enough. I don't know Docker well enough to dig into this I'm afraid. I can only suggest looking for differences in the runc versions between Docker and podman and/or experimenting with different Docker storage drivers. It would be useful to the community to understand where the problem lies but I can't really investigate without a reproducer.

[janl]

please reopen if there is something CouchDB can do to run better in the docker env