Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Couchdb stops activity and then after some time we have always timeouts #3083

Closed
raulmartinezr opened this issue Aug 17, 2020 · 8 comments
Closed

Comments

@raulmartinezr
Copy link

Description

We use dockerized couchdb in local development environment to test applications.
The procedure for each test:

  • create a database in couchdb
  • test whatever we need to test inside this DB
  • delete it in case it's not necessary anymore

We have aprox 70 tests

For each database a few documents are created

  • less than 10 documents usually (max 50 when we test indexes created by ddoc)
  • 2 design documents, one with two mango indexes and the other one with 1 mango index

If we execute tests module by module (each module contain between 1 and 10 tests ) there is not any issue, but if we execute all at the same time (sequentially, without parallelism), then at some point tests hangs

We observed

  • Couchdb stops writting logs (CPU consumption decreases)
  • We still have responses to quieries for some time, 1min aprox
  • Then we have not responses from couchdb anymore

Example (It's not always the same)

  • Last logs written (querying a view with /master/_partition/iam/_find ). After that CPU consumption of couchdb goes down.
[debug] 2020-08-17T19:30:05.663402Z nonode@nohost <0.20917.0> a13fe062bc no record of user admin
[debug] 2020-08-17T19:30:05.663457Z nonode@nohost <0.20917.0> a13fe062bc timeout 600
[debug] 2020-08-17T19:30:05.663495Z nonode@nohost <0.20917.0> a13fe062bc Successful cookie auth as: "admin"
[notice] 2020-08-17T19:30:05.665187Z nonode@nohost <0.20917.0> a13fe062bc 127.0.0.1:5984 172.28.0.1 admin POST /master/_partition/iam/_find 200 ok 2
[debug] 2020-08-17T19:30:05.677477Z nonode@nohost <0.20917.0> be296403e4 no record of user admin
[debug] 2020-08-17T19:30:05.677539Z nonode@nohost <0.20917.0> be296403e4 timeout 600
[debug] 2020-08-17T19:30:05.677570Z nonode@nohost <0.20917.0> be296403e4 Successful cookie auth as: "admin"
  • Request captured with tcpdump
    image

  • After some time (1min 9s), timeouts
    image

HW and environment

  • Host machine runs Ubuntu 20.04, with 8cores and 16Gb RAM and 512GB SSD (250free)
  • Docker container has not any limitation in CPU/Memory/Space
  • Couchdb configured as single node (full configuration below)

This is how the container processes looks. Earlang (beam.smp) is the most consuming, with peaks of 70% CPU
image

Theads of beam.smp: Just when issue happens
I monitored threads during the whole process. Just before the crash, it seems scheduler threads increase the activity

top - 20:55:05 up  2:58,  0 users,  load average: 3.93, 2.31, 2.35
Threads:  46 total,   0 running,  46 sleeping,   0 stopped,   0 zombie
%Cpu(s): 46.2 us,  4.3 sy,  0.0 ni, 49.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15687.8 total,   1947.6 free,   7294.3 used,   6445.9 buff/cache
MiB Swap:    980.0 total,    980.0 free,      0.0 used.   7200.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     54 couchdb   20   0 4568544  62092  10904 S  40.0   0.4   0:07.25 2_scheduler
     53 couchdb   20   0 4568544  62092  10904 S  33.3   0.4   0:20.74 1_scheduler
     55 couchdb   20   0 4568544  62092  10904 S  33.3   0.4   0:06.63 3_scheduler
     56 couchdb   20   0 4568544  62092  10904 S  20.0   0.4   0:05.66 4_scheduler
     37 couchdb   20   0 4568544  62092  10904 S   6.7   0.4   0:00.03 async_2
      6 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.03 beam.smp
     34 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.00 sys_sig_dispatc
     35 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.00 sys_msg_dispatc
     36 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.18 async_1
     38 couchdb   20   0 4568544  62092  10904 S   0.0   0.4   0:00.14 async_3
.....

Any idea about what could be the cause? Any hint would be appreaciated.

Steps to Reproduce

There is not a fixed trigger for the issue.

Expected Behaviour

We would expect couchdb can handle this load even with docker. It's not heavy, during tests we have 15operations per second max.

Your Environment

{"couchdb":"Welcome","version":"3.1.0","git_sha":"ff0feea20","uuid":"b17edbd1de7d1504022d6f359ff9a4f8","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
  • CouchDB version used: 3.1.0
  • Browser name and version: Not relevant
  • Operating system and version: Couchdb docker@Ubuntu20.04

Additional Context

Full couchdb configuration

Configuration Settings:
  [admins] admin="******"
  [attachments] compressible_types="text/*, application/javascript, application/json, application/xml"
  [attachments] compression_level="8"
  [chttpd] backlog="512"
  [chttpd] bind_address="any"
  [chttpd] max_db_number_for_dbs_info_req="100"
  [chttpd] port="5984"
  [chttpd] prefer_minimal="Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary"
  [chttpd] require_valid_user="false"
  [chttpd] server_options="[{backlog, 512}, {acceptor_pool_size, 64}, {max, 4096}]"
  [chttpd] socket_options="[{sndbuf, 262144}, {nodelay, true}]"
  [cluster] n="3"
  [cluster] q="2"
  [cors] credentials="false"
  [couch_httpd_auth] allow_persistent_cookies="true"
  [couch_httpd_auth] auth_cache_size="50"
  [couch_httpd_auth] authentication_db="_users"
  [couch_httpd_auth] authentication_redirect="/_utils/session.html"
  [couch_httpd_auth] iterations="10"
  [couch_httpd_auth] require_valid_user="false"
  [couch_httpd_auth] secret="00464db7ba8beb6a5915e4f5dbd03a49"
  [couch_httpd_auth] timeout="600"
  [couch_peruser] database_prefix="userdb-"
  [couch_peruser] delete_dbs="false"
  [couch_peruser] enable="false"
  [couchdb] attachment_stream_buffer_size="4096"
  [couchdb] changes_doc_ids_optimization_threshold="100"
  [couchdb] database_dir="./data"
  [couchdb] default_engine="couch"
  [couchdb] default_security="admin_only"
  [couchdb] file_compression="snappy"
  [couchdb] max_dbs_open="10000"
  [couchdb] max_document_size="8000000"
  [couchdb] os_process_timeout="20000"
  [couchdb] single_node="true"
  [couchdb] users_db_security_editable="false"
  [couchdb] uuid="b17edbd1de7d1504022d6f359ff9a4f8"
  [couchdb] view_index_dir="./data"
  [couchdb_engines] couch="couch_bt_engine"
  [csp] enable="true"
  [fabric] request_timeout="infinity"
  [feature_flags] partitioned||*="true"
  [httpd] allow_jsonp="false"
  [httpd] authentication_handlers="{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}"
  [httpd] bind_address="127.0.0.1"
  [httpd] enable_cors="false"
  [httpd] enable_xframe_options="false"
  [httpd] max_http_request_size="4294967296"
  [httpd] port="5986"
  [httpd] secure_rewrites="true"
  [httpd] socket_options="[{sndbuf, 262144}]"
  [indexers] couch_mrview="true"
  [ioq] concurrency="10"
  [ioq] ratio="0.01"
  [ioq.bypass] compaction="false"
  [ioq.bypass] os_process="true"
  [ioq.bypass] read="true"
  [ioq.bypass] shard_sync="false"
  [ioq.bypass] view_update="true"
  [ioq.bypass] write="true"
  [log] level="debug"
  [log] writer="stderr"
  [query_server_config] os_process_limit="2000"
  [query_server_config] os_process_soft_limit="1000"
  [query_server_config] reduce_limit="true"
  [replicator] connection_timeout="30000"
  [replicator] http_connections="20"
  [replicator] interval="60000"
  [replicator] max_churn="20"
  [replicator] max_jobs="500"
  [replicator] retries_per_request="5"
  [replicator] socket_options="[{keepalive, true}, {nodelay, false}]"
  [replicator] ssl_certificate_max_depth="3"
  [replicator] startup_jitter="5000"
  [replicator] verify_ssl_certificates="false"
  [replicator] worker_batch_size="500"
  [replicator] worker_processes="4"
  [ssl] port="6984"
  [uuids] algorithm="sequential"
  [uuids] max_count="1000"
  [vendor] name="The Apache Software Foundation"

Some errors identified in the startup. They seem not relevant for the case.

  • _users db does not exists (but seems to be created afterwards)
[error] 2020-08-17T19:18:47.731011Z nonode@nohost emulator -------- Error in process <0.372.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,374}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}
  • I suppose without any effect as long as it's configured as single node [couchdb] single_node="true"
[error] 2020-08-17T19:18:47.799365Z nonode@nohost <0.457.0> -------- Request to create N=3 DB but only 1 node(s)
[error] 2020-08-17T19:18:47.812920Z nonode@nohost <0.457.0> -------- Request to create N=3 DB but only 1 node(s)
@raulmartinezr raulmartinezr changed the title Couchdb stops writting logs and and then after some time we have always timeouts Couchdb stops activity and then after some time we have always timeouts Aug 17, 2020
@willholley
Copy link
Member

@raulmartinezr are you using Docker volumes to mount a CouchDB data volume or using a storage driver to write to the Docker filesystem directly? If the latter, which driver are you using?

@raulmartinezr
Copy link
Author

Hi @willholley !

we are using tmfs

    tmpfs:
      - /opt/couchdb/data

As long as we do not need to persist data between runs, that option was usefull.

BR,
Raul

@willholley
Copy link
Member

Thanks @raulmartinezr. Just out of interest, are you able to reproduce the problem using podman instead of Docker? Afaik podman will use the same container runtime (runc) as Docker so it would help narrow down the problem to something Docker-specific vs a container/CouchDB issue.

@raulmartinezr
Copy link
Author

Hi @willholley ,

I tried to reproduce with podman but it was not possible. With podman it's working fine. Seems to be something related to docker daemon.

What do you think?

Thanks,
Raul

@willholley
Copy link
Member

Thanks @raulmartinezr - that's certainly interesting. What version of Docker are you running (output of docker version would be helpful)?

@raulmartinezr
Copy link
Author

Hello!

Thats what I have

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:45:44 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:44:15 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

@willholley
Copy link
Member

@raulmartinezr that looks recent enough. I don't know Docker well enough to dig into this I'm afraid. I can only suggest looking for differences in the runc versions between Docker and podman and/or experimenting with different Docker storage drivers. It would be useful to the community to understand where the problem lies but I can't really investigate without a reproducer.

@janl
Copy link
Member

janl commented Mar 14, 2021

please reopen if there is something CouchDB can do to run better in the docker env

@janl janl closed this as completed Mar 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants