New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CouchDB 2.1 crashes when a mount point contains whitespace #732

Closed
obi458 opened this Issue Aug 1, 2017 · 31 comments

Comments

Projects
None yet
8 participants
@obi458

obi458 commented Aug 1, 2017

When time machine runs on the couchdb data folder the compaction failed and couchdb shutdown:

I added couch_log to see why, this gives me the path "/Volumes/Time"!!!!!:

abs_path(Path0) ->
    couch_log:debug("-------------------- abs_path ~p", [Path0]),
    {ok, Info} = file:read_link_info(Path0),
    case Info#file_info.type of
        symlink ->
            {ok, Path} = file:read_link(Path0),
            abs_path(Path);
        _ ->
            abs_path2(Path0)
    end.

and gives:

[debug] 2017-08-01T08:24:39.917976Z couchdb@localhost <0.2442.0> -------- Supervisor couch_secondary_services started couch_compaction_daemon:start_link() at pid <0.2682.0>
[debug] 2017-08-01T08:24:39.926015Z couchdb@localhost <0.2684.0> -------- Fragmentation for database `_dbs` is 0%, estimated space for compaction is 28864 bytes.
[debug] 2017-08-01T08:24:39.926312Z couchdb@localhost <0.2684.0> -------- Fragmentation for database `_nodes` is 0%, estimated space for compaction is 8372 bytes.
[debug] 2017-08-01T08:24:39.926747Z couchdb@localhost <0.2684.0> -------- Fragmentation for database `_replicator` is 0%, estimated space for compaction is 8378 bytes.
[debug] 2017-08-01T08:24:39.927227Z couchdb@localhost <0.2684.0> -------- Fragmentation for view group `_replicator` (database `_replicator`) is 0%, estimated space for compaction is 51 bytes.
[debug] 2017-08-01T08:24:39.927522Z couchdb@localhost <0.2684.0> -------- Fragmentation for database `_users` is 0%, estimated space for compaction is 8378 bytes.
[debug] 2017-08-01T08:24:39.927944Z couchdb@localhost <0.2684.0> -------- Fragmentation for view group `_auth` (database `_users`) is 0%, estimated space for compaction is 51 bytes.
[debug] 2017-08-01T08:24:39.928793Z couchdb@localhost <0.2684.0> -------- Fragmentation for database `shards/00000000-1fffffff/_global_changes.1501572189` is 98%, estimated space for compaction is 13246 bytes.
[debug] 2017-08-01T08:24:39.928851Z couchdb@localhost <0.2684.0> -------- -------------------- abs_path "./data"
[debug] 2017-08-01T08:24:39.928991Z couchdb@localhost <0.2684.0> -------- -------------------- abs_path "/Volumes/Time"
[error] 2017-08-01T08:24:39.929252Z couchdb@localhost emulator -------- Error in process <0.2684.0> on node couchdb@localhost with exit value:
{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,527}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,273}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,195}]}]}
@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Aug 1, 2017

Member

Can you please provide which version of CouchDB you are using, your local.ini file, the path to where you installed CouchDB? Thanks.

Member

wohali commented Aug 1, 2017

Can you please provide which version of CouchDB you are using, your local.ini file, the path to where you installed CouchDB? Thanks.

@wohali wohali added the macos label Aug 1, 2017

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 1, 2017

  1. I build it with the latest master(10 hours ago): couch-2.1.0-RC1-4-gf8cb6f97c
  2. Path is rel/couchdb where it is after make release
  3. local.ini:

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings
; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.

[couchdb]
;max_document_size = 4294967296 ; bytes
;os_process_timeout = 5000
uuid = 039e269bb0d0bec878ad366eb24b37b1

[couch_peruser]
; If enabled, couch_peruser ensures that a private per-user database
; exists for each document in _users. These databases are writable only
; by the corresponding user. Databases are in the following form:
; userdb-{hex encoded username}
;enable = true
; If set to true and a user is deleted, the respective database gets
; deleted as well.
;delete_dbs = true

[chttpd]
port = 5983
;bind_address = 127.0.0.1
; Options for the MochiWeb HTTP server.
;server_options = [{backlog, 128}, {acceptor_pool_size, 16}]
; For more socket options, consult Erlang's module 'inet' man page.
;socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]
bind_address = 0.0.0.0
require_valid_user = true
proxy_use_secret = true
authentication_handler =
authentication_handlers = {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}

[httpd]
port = 5982
; NOTE that this only configures the "backend" node-local port, not the
; "frontend" clustered port. You probably don't want to change anything in
; this section.
; Uncomment next line to trigger basic-auth popup on unauthorized requests.
;WWW-Authenticate = Basic realm="administrator"

; Uncomment next line to set the configuration modification whitelist. Only
; whitelisted values may be changed via the /_config URLs. To allow the admin
; to change this value over HTTP, remember to include {httpd,config_whitelist}
; itself. Excluding it from the list would require editing this file to update
; the whitelist.
;config_whitelist = [{httpd,config_whitelist}, {log,level}, {etc,etc}]
authentication_handlers = {couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}

[query_servers]
;nodejs = /usr/local/bin/couchjs-node /path/to/couchdb/share/server/main.js

[httpd_global_handlers]
;_google = {couch_httpd_proxy, handle_proxy_req, <<"http://www.google.com">>}

[couch_httpd_auth]
; If you set this to true, you should also uncomment the WWW-Authenticate line
; above. If you don't configure a WWW-Authenticate header, CouchDB will send
; Basic realm="server" in order to prevent you getting logged out.
; require_valid_user = false
secret = 59edb32b73e57cda6be862966548e535
require_valid_user = true
proxy_use_secret = true

[os_daemons]
; For any commands listed here, CouchDB will attempt to ensure that
; the process remains alive. Daemons should monitor their environment
; to know when to exit. This can most easily be accomplished by exiting
; when stdin is closed.
;foo = /path/to/command -with args

[daemons]
; enable SSL support by uncommenting the following line and supply the PEM's below.
; the default ssl port CouchDB listens on is 6984
; httpsd = {chttpd, start_link, [https]}

[ssl]
;cert_file = /full/path/to/server_cert.pem
;key_file = /full/path/to/server_key.pem
;password = somepassword
; set to true to validate peer certificates
;verify_ssl_certificates = false
; Set to true to fail if the client does not send a certificate. Only used if verify_ssl_certificates is true.
;fail_if_no_peer_cert = false
; Path to file containing PEM encoded CA certificates (trusted
; certificates used for verifying a peer certificate). May be omitted if
; you do not want to verify the peer.
;cacert_file = /full/path/to/cacertf
; The verification fun (optional) if not specified, the default
; verification fun will be used.
;verify_fun = {Module, VerifyFun}
; maximum peer certificate depth
;ssl_certificate_max_depth = 1
;
; Reject renegotiations that do not live up to RFC 5746.
;secure_renegotiate = true
; The cipher suites that should be supported.
; Can be specified in erlang format "{ecdhe_ecdsa,aes_128_cbc,sha256}"
; or in OpenSSL format "ECDHE-ECDSA-AES128-SHA256".
;ciphers = ["ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA"]
; The SSL/TLS versions to support
;tls_versions = [tlsv1, 'tlsv1.1', 'tlsv1.2']

; To enable Virtual Hosts in CouchDB, add a vhost = path directive. All requests to
; the Virual Host will be redirected to the path. In the example below all requests
; to http://example.com/ are redirected to /database.
; If you run CouchDB on a specific port, include the port number in the vhost:
; example.com:5984 = /database
[vhosts]
;example.com = /database/

[update_notification]
;unique notifier name=/full/path/to/exe -with "cmd line arg"

; To create an admin account uncomment the '[admins]' section below and add a
; line in the format 'username = password'. When you next start CouchDB, it
; will change the password to a hash (so that your passwords don't linger
; around in plain-text files). You can add more admin accounts with more
; 'username = password' lines. Don't forget to restart CouchDB after
; changing this.
[admins]
;admin = mysecretpassword
admin = -pbkdf2-516d1745da6a942e0075ce147bd8fbee59143d56,93424e9d5f2013197d02b47acdd18e29,10

[cluster]
n = 1

[native_query_servers]
erlang = {couch_native_process, start_link, []}

[vendor]

[log]
level = debug

[compactions]
data = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]
client = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]
chat = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]

[replicator]
max_replication_retry_count = infinity

obi458 commented Aug 1, 2017

  1. I build it with the latest master(10 hours ago): couch-2.1.0-RC1-4-gf8cb6f97c
  2. Path is rel/couchdb where it is after make release
  3. local.ini:

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings
; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.

[couchdb]
;max_document_size = 4294967296 ; bytes
;os_process_timeout = 5000
uuid = 039e269bb0d0bec878ad366eb24b37b1

[couch_peruser]
; If enabled, couch_peruser ensures that a private per-user database
; exists for each document in _users. These databases are writable only
; by the corresponding user. Databases are in the following form:
; userdb-{hex encoded username}
;enable = true
; If set to true and a user is deleted, the respective database gets
; deleted as well.
;delete_dbs = true

[chttpd]
port = 5983
;bind_address = 127.0.0.1
; Options for the MochiWeb HTTP server.
;server_options = [{backlog, 128}, {acceptor_pool_size, 16}]
; For more socket options, consult Erlang's module 'inet' man page.
;socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]
bind_address = 0.0.0.0
require_valid_user = true
proxy_use_secret = true
authentication_handler =
authentication_handlers = {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}

[httpd]
port = 5982
; NOTE that this only configures the "backend" node-local port, not the
; "frontend" clustered port. You probably don't want to change anything in
; this section.
; Uncomment next line to trigger basic-auth popup on unauthorized requests.
;WWW-Authenticate = Basic realm="administrator"

; Uncomment next line to set the configuration modification whitelist. Only
; whitelisted values may be changed via the /_config URLs. To allow the admin
; to change this value over HTTP, remember to include {httpd,config_whitelist}
; itself. Excluding it from the list would require editing this file to update
; the whitelist.
;config_whitelist = [{httpd,config_whitelist}, {log,level}, {etc,etc}]
authentication_handlers = {couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}

[query_servers]
;nodejs = /usr/local/bin/couchjs-node /path/to/couchdb/share/server/main.js

[httpd_global_handlers]
;_google = {couch_httpd_proxy, handle_proxy_req, <<"http://www.google.com">>}

[couch_httpd_auth]
; If you set this to true, you should also uncomment the WWW-Authenticate line
; above. If you don't configure a WWW-Authenticate header, CouchDB will send
; Basic realm="server" in order to prevent you getting logged out.
; require_valid_user = false
secret = 59edb32b73e57cda6be862966548e535
require_valid_user = true
proxy_use_secret = true

[os_daemons]
; For any commands listed here, CouchDB will attempt to ensure that
; the process remains alive. Daemons should monitor their environment
; to know when to exit. This can most easily be accomplished by exiting
; when stdin is closed.
;foo = /path/to/command -with args

[daemons]
; enable SSL support by uncommenting the following line and supply the PEM's below.
; the default ssl port CouchDB listens on is 6984
; httpsd = {chttpd, start_link, [https]}

[ssl]
;cert_file = /full/path/to/server_cert.pem
;key_file = /full/path/to/server_key.pem
;password = somepassword
; set to true to validate peer certificates
;verify_ssl_certificates = false
; Set to true to fail if the client does not send a certificate. Only used if verify_ssl_certificates is true.
;fail_if_no_peer_cert = false
; Path to file containing PEM encoded CA certificates (trusted
; certificates used for verifying a peer certificate). May be omitted if
; you do not want to verify the peer.
;cacert_file = /full/path/to/cacertf
; The verification fun (optional) if not specified, the default
; verification fun will be used.
;verify_fun = {Module, VerifyFun}
; maximum peer certificate depth
;ssl_certificate_max_depth = 1
;
; Reject renegotiations that do not live up to RFC 5746.
;secure_renegotiate = true
; The cipher suites that should be supported.
; Can be specified in erlang format "{ecdhe_ecdsa,aes_128_cbc,sha256}"
; or in OpenSSL format "ECDHE-ECDSA-AES128-SHA256".
;ciphers = ["ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA"]
; The SSL/TLS versions to support
;tls_versions = [tlsv1, 'tlsv1.1', 'tlsv1.2']

; To enable Virtual Hosts in CouchDB, add a vhost = path directive. All requests to
; the Virual Host will be redirected to the path. In the example below all requests
; to http://example.com/ are redirected to /database.
; If you run CouchDB on a specific port, include the port number in the vhost:
; example.com:5984 = /database
[vhosts]
;example.com = /database/

[update_notification]
;unique notifier name=/full/path/to/exe -with "cmd line arg"

; To create an admin account uncomment the '[admins]' section below and add a
; line in the format 'username = password'. When you next start CouchDB, it
; will change the password to a hash (so that your passwords don't linger
; around in plain-text files). You can add more admin accounts with more
; 'username = password' lines. Don't forget to restart CouchDB after
; changing this.
[admins]
;admin = mysecretpassword
admin = -pbkdf2-516d1745da6a942e0075ce147bd8fbee59143d56,93424e9d5f2013197d02b47acdd18e29,10

[cluster]
n = 1

[native_query_servers]
erlang = {couch_native_process, start_link, []}

[vendor]

[log]
level = debug

[compactions]
data = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]
client = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]
chat = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]

[replicator]
max_replication_retry_count = infinity

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 1, 2017

Here is my temporarily workaround!

abs_path(Path0) ->
% if we get an error use abs_path2
case file:read_link_info(Path0) of
{ok, Info} -> case Info#file_info.type of
symlink ->
{ok, Path} = file:read_link(Path0),
abs_path(Path);
_ ->
abs_path2(Path0)
end;
{error,_Reason} -> abs_path2(Path0)
end.

obi458 commented Aug 1, 2017

Here is my temporarily workaround!

abs_path(Path0) ->
% if we get an error use abs_path2
case file:read_link_info(Path0) of
{ok, Info} -> case Info#file_info.type of
symlink ->
{ok, Path} = file:read_link(Path0),
abs_path(Path);
_ ->
abs_path2(Path0)
end;
{error,_Reason} -> abs_path2(Path0)
end.

@janl

This comment has been minimized.

Show comment
Hide comment
@janl

janl Aug 3, 2017

Member

The ini doesn’t show the database_dir and view_index_dir directives.

Member

janl commented Aug 3, 2017

The ini doesn’t show the database_dir and view_index_dir directives.

@janl

This comment has been minimized.

Show comment
Hide comment
@janl

janl Aug 3, 2017

Member

in fact, can you dump /_node/<name>/_config here as well. And share ps ax | grep beam.

Member

janl commented Aug 3, 2017

in fact, can you dump /_node/<name>/_config here as well. And share ps ax | grep beam.

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 4, 2017

Dirs from default.ini:
database_dir = ./data
view_index_dir = ./data

obi458 commented Aug 4, 2017

Dirs from default.ini:
database_dir = ./data
view_index_dir = ./data

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 4, 2017

single node therefore no /_node/...

obi458 commented Aug 4, 2017

single node therefore no /_node/...

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 4, 2017

77462 s006 S+ 153:01.58 /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/../erts-8.3/bin/beam.smp -K true -A 16 -Bd -- -root /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/.. -progname couchdb -- -home /Users/obi458 -- -boot /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/../releases/2.1.0/couchdb -name couchdb1@127.0.0.1 -setcookie monster -kernel error_logger silent -sasl sasl_error_logger false -noshell -noinput -config /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/../releases/2.1.0/sys.config

obi458 commented Aug 4, 2017

77462 s006 S+ 153:01.58 /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/../erts-8.3/bin/beam.smp -K true -A 16 -Bd -- -root /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/.. -progname couchdb -- -home /Users/obi458 -- -boot /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/../releases/2.1.0/couchdb -name couchdb1@127.0.0.1 -setcookie monster -kernel error_logger silent -sasl sasl_error_logger false -noshell -noinput -config /Users/obi458/devel/couchdb_latest/rel/dev1/couchdb/bin/../releases/2.1.0/sys.config

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 4, 2017

I guess this is a race, race between get_disk_data() and free_space_rec(abs_path(Path), DiskData) cause
get_disk_data()(From Help): The function is asynchronous in the sense that it does not invoke a disk check, but returns the latest available value.
file:read_link_info(Name): Name must exists!!

obi458 commented Aug 4, 2017

I guess this is a race, race between get_disk_data() and free_space_rec(abs_path(Path), DiskData) cause
get_disk_data()(From Help): The function is asynchronous in the sense that it does not invoke a disk check, but returns the latest available value.
file:read_link_info(Name): Name must exists!!

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 4, 2017

Hint, just called:

18> os_mon:call(disksup, get_disk_data, infinity).
[{"/",3033388032,38},{"/Volumes/Time",2927833480,100}]
19> file:read_link_info("/Volumes/Time").
{error,enoent}

obi458 commented Aug 4, 2017

Hint, just called:

18> os_mon:call(disksup, get_disk_data, infinity).
[{"/",3033388032,38},{"/Volumes/Time",2927833480,100}]
19> file:read_link_info("/Volumes/Time").
{error,enoent}

@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Aug 4, 2017

Member

You say "single node therefore no /_node" but that's not true. There is always a /_node/<name>@<host>/_config endpoint. Check your vm.args file for the node's default name; it's something like couchdb@127.0.0.1 or noname@nohost.

Member

wohali commented Aug 4, 2017

You say "single node therefore no /_node" but that's not true. There is always a /_node/<name>@<host>/_config endpoint. Check your vm.args file for the node's default name; it's something like couchdb@127.0.0.1 or noname@nohost.

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 4, 2017

Can not find /_node/couchdb1@127.0.0.1/_config, what does it mean absolute relative path or something else, i don't know?

obi458 commented Aug 4, 2017

Can not find /_node/couchdb1@127.0.0.1/_config, what does it mean absolute relative path or something else, i don't know?

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 4, 2017

OK, you mean http://...
Here:

{"native_query_servers":{"erlang":"{couch_native_process, start_link, []}","query":"{mango_native_proc, start_link, []}"},"httpd_design_handlers":{"_compact":"{couch_mrview_http, handle_compact_req}","_info":"{couch_mrview_http, handle_info_req}","_list":"{couch_mrview_show, handle_view_list_req}","_rewrite":"{couch_httpd_rewrite, handle_rewrite_req}","_show":"{couch_mrview_show, handle_doc_show_req}","_update":"{couch_mrview_show, handle_doc_update_req}","_view":"{couch_mrview_http, handle_view_req}","_view_changes":"{couch_mrview_http, handle_view_changes_req}"},"uuids":{"algorithm":"sequential","max_count":"1000"},"cluster":{"n":"1","q":"8"},"cors":{"credentials":"false"},"chttpd":{"authentication_handlers":"{chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}","backlog":"512","bind_address":"0.0.0.0","docroot":"./share/www","port":"5983","proxy_use_secret":"true","require_valid_user":"true","socket_options":"[{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]"},"httpd_global_handlers":{"/":"{couch_httpd_misc_handlers, handle_welcome_req, <<"Welcome">>}","_active_tasks":"{couch_httpd_misc_handlers, handle_task_status_req}","_all_dbs":"{couch_httpd_misc_handlers, handle_all_dbs_req}","_config":"{couch_httpd_misc_handlers, handle_config_req}","_plugins":"{couch_plugins_httpd, handle_req}","_replicate":"{couch_replicator_httpd, handle_req}","_restart":"{couch_httpd_misc_handlers, handle_restart_req}","_session":"{couch_httpd_auth, handle_session_req}","_stats":"{couch_stats_httpd, handle_stats_req}","_system":"{chttpd_misc, handle_system_req}","_utils":"{couch_httpd_misc_handlers, handle_utils_dir_req, "./share/www"}","_uuids":"{couch_httpd_misc_handlers, handle_uuids_req}","favicon.ico":"{couch_httpd_misc_handlers, handle_favicon_req, "./share/www"}"},"attachments":{"compressible_types":"text/*, application/javascript, application/json, application/xml","compression_level":"8"},"admins":{"admin":"-pbkdf2-e2f272537f56b0254d1397e37425d73f94cd3979,7a54c5ca3cb97898aba94a4e689a4eaf,10"},"query_server_config":{"os_process_limit":"100","reduce_limit":"true"},"vendor":{"name":"The Apache Software Foundation"},"replicator":{"connection_timeout":"30000","http_connections":"20","interval":"60000","max_churn":"20","max_jobs":"500","max_replication_retry_count":"infinity","retries_per_request":"10","socket_options":"[{keepalive, true}, {nodelay, false}]","ssl_certificate_max_depth":"3","startup_jitter":"5000","verify_ssl_certificates":"false","worker_batch_size":"500","worker_processes":"4"},"ssl":{"port":"6984"},"log":{"level":"error","writer":"stderr"},"indexers":{"couch_mrview":"true"},"view_compaction":{"keyvalue_buffer_size":"2097152"},"query_servers":{"coffeescript":"./bin/couchjs ./share/server/main-coffee.js","javascript":"./bin/couchjs ./share/server/main.js"},"features":{"scheduler":"true"},"daemons":{"auth_cache":"{couch_auth_cache, start_link, []}","compaction_daemon":"{couch_compaction_daemon, start_link, []}","couch_peruser":"{couch_peruser, start_link, []}","external_manager":"{couch_external_manager, start_link, []}","httpd":"{couch_httpd, start_link, []}","index_server":"{couch_index_server, start_link, []}","os_daemons":"{couch_os_daemons, start_link, []}","query_servers":"{couch_proc_manager, start_link, []}","uuids":"{couch_uuids, start, []}","vhosts":"{couch_httpd_vhost, start_link, []}"},"couch_peruser":{"delete_dbs":"false","enable":"false"},"httpd":{"allow_jsonp":"false","authentication_handlers":"{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}","bind_address":"0.0.0.0","default_handler":"{couch_httpd_db, handle_request}","enable_cors":"false","enable_xframe_options":"false","max_http_request_size":"67108864","port":"5982","secure_rewrites":"true","socket_options":"[{recbuf, 262144}, {sndbuf, 262144}]","vhost_global_handlers":"_utils, _uuids, _session, _users"},"httpd_db_handlers":{"_all_docs":"{couch_mrview_http, handle_all_docs_req}","_changes":"{couch_httpd_db, handle_db_changes_req}","_compact":"{couch_httpd_db, handle_compact_req}","_design":"{couch_httpd_db, handle_design_req}","_design_docs":"{couch_mrview_http, handle_design_docs_req}","_local_docs":"{couch_mrview_http, handle_local_docs_req}","_temp_view":"{couch_mrview_http, handle_temp_view_req}","_view_cleanup":"{couch_mrview_http, handle_cleanup_req}"},"database_compaction":{"checkpoint_after":"5242880","doc_buffer_size":"524288"},"csp":{"enable":"true"},"couch_httpd_auth":{"allow_persistent_cookies":"false","auth_cache_size":"50","authentication_db":"_users","authentication_redirect":"/_utils/session.html","iterations":"10","proxy_use_secret":"true","require_valid_user":"true","secret":"59edb32b73e57cda6be862966548e535","timeout":"600"},"couchdb":{"attachment_stream_buffer_size":"4096","changes_doc_ids_optimization_threshold":"100","database_dir":"./data","default_security":"admin_local","delayed_commits":"false","file_compression":"snappy","max_dbs_open":"500","os_process_timeout":"5000","uuid":"039e269bb0d0bec878ad366eb24b37b1","view_index_dir":"./data"},"compactions":{"_default":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}]","bi_chat":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","bi_client_info":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","bi_data":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","chat":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","client":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","data":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]"},"compaction_daemon":{"check_interval":"300","min_file_size":"131072"}}

obi458 commented Aug 4, 2017

OK, you mean http://...
Here:

{"native_query_servers":{"erlang":"{couch_native_process, start_link, []}","query":"{mango_native_proc, start_link, []}"},"httpd_design_handlers":{"_compact":"{couch_mrview_http, handle_compact_req}","_info":"{couch_mrview_http, handle_info_req}","_list":"{couch_mrview_show, handle_view_list_req}","_rewrite":"{couch_httpd_rewrite, handle_rewrite_req}","_show":"{couch_mrview_show, handle_doc_show_req}","_update":"{couch_mrview_show, handle_doc_update_req}","_view":"{couch_mrview_http, handle_view_req}","_view_changes":"{couch_mrview_http, handle_view_changes_req}"},"uuids":{"algorithm":"sequential","max_count":"1000"},"cluster":{"n":"1","q":"8"},"cors":{"credentials":"false"},"chttpd":{"authentication_handlers":"{chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}","backlog":"512","bind_address":"0.0.0.0","docroot":"./share/www","port":"5983","proxy_use_secret":"true","require_valid_user":"true","socket_options":"[{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]"},"httpd_global_handlers":{"/":"{couch_httpd_misc_handlers, handle_welcome_req, <<"Welcome">>}","_active_tasks":"{couch_httpd_misc_handlers, handle_task_status_req}","_all_dbs":"{couch_httpd_misc_handlers, handle_all_dbs_req}","_config":"{couch_httpd_misc_handlers, handle_config_req}","_plugins":"{couch_plugins_httpd, handle_req}","_replicate":"{couch_replicator_httpd, handle_req}","_restart":"{couch_httpd_misc_handlers, handle_restart_req}","_session":"{couch_httpd_auth, handle_session_req}","_stats":"{couch_stats_httpd, handle_stats_req}","_system":"{chttpd_misc, handle_system_req}","_utils":"{couch_httpd_misc_handlers, handle_utils_dir_req, "./share/www"}","_uuids":"{couch_httpd_misc_handlers, handle_uuids_req}","favicon.ico":"{couch_httpd_misc_handlers, handle_favicon_req, "./share/www"}"},"attachments":{"compressible_types":"text/*, application/javascript, application/json, application/xml","compression_level":"8"},"admins":{"admin":"-pbkdf2-e2f272537f56b0254d1397e37425d73f94cd3979,7a54c5ca3cb97898aba94a4e689a4eaf,10"},"query_server_config":{"os_process_limit":"100","reduce_limit":"true"},"vendor":{"name":"The Apache Software Foundation"},"replicator":{"connection_timeout":"30000","http_connections":"20","interval":"60000","max_churn":"20","max_jobs":"500","max_replication_retry_count":"infinity","retries_per_request":"10","socket_options":"[{keepalive, true}, {nodelay, false}]","ssl_certificate_max_depth":"3","startup_jitter":"5000","verify_ssl_certificates":"false","worker_batch_size":"500","worker_processes":"4"},"ssl":{"port":"6984"},"log":{"level":"error","writer":"stderr"},"indexers":{"couch_mrview":"true"},"view_compaction":{"keyvalue_buffer_size":"2097152"},"query_servers":{"coffeescript":"./bin/couchjs ./share/server/main-coffee.js","javascript":"./bin/couchjs ./share/server/main.js"},"features":{"scheduler":"true"},"daemons":{"auth_cache":"{couch_auth_cache, start_link, []}","compaction_daemon":"{couch_compaction_daemon, start_link, []}","couch_peruser":"{couch_peruser, start_link, []}","external_manager":"{couch_external_manager, start_link, []}","httpd":"{couch_httpd, start_link, []}","index_server":"{couch_index_server, start_link, []}","os_daemons":"{couch_os_daemons, start_link, []}","query_servers":"{couch_proc_manager, start_link, []}","uuids":"{couch_uuids, start, []}","vhosts":"{couch_httpd_vhost, start_link, []}"},"couch_peruser":{"delete_dbs":"false","enable":"false"},"httpd":{"allow_jsonp":"false","authentication_handlers":"{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler},{couch_httpd_auth, proxy_authentication_handler}","bind_address":"0.0.0.0","default_handler":"{couch_httpd_db, handle_request}","enable_cors":"false","enable_xframe_options":"false","max_http_request_size":"67108864","port":"5982","secure_rewrites":"true","socket_options":"[{recbuf, 262144}, {sndbuf, 262144}]","vhost_global_handlers":"_utils, _uuids, _session, _users"},"httpd_db_handlers":{"_all_docs":"{couch_mrview_http, handle_all_docs_req}","_changes":"{couch_httpd_db, handle_db_changes_req}","_compact":"{couch_httpd_db, handle_compact_req}","_design":"{couch_httpd_db, handle_design_req}","_design_docs":"{couch_mrview_http, handle_design_docs_req}","_local_docs":"{couch_mrview_http, handle_local_docs_req}","_temp_view":"{couch_mrview_http, handle_temp_view_req}","_view_cleanup":"{couch_mrview_http, handle_cleanup_req}"},"database_compaction":{"checkpoint_after":"5242880","doc_buffer_size":"524288"},"csp":{"enable":"true"},"couch_httpd_auth":{"allow_persistent_cookies":"false","auth_cache_size":"50","authentication_db":"_users","authentication_redirect":"/_utils/session.html","iterations":"10","proxy_use_secret":"true","require_valid_user":"true","secret":"59edb32b73e57cda6be862966548e535","timeout":"600"},"couchdb":{"attachment_stream_buffer_size":"4096","changes_doc_ids_optimization_threshold":"100","database_dir":"./data","default_security":"admin_local","delayed_commits":"false","file_compression":"snappy","max_dbs_open":"500","os_process_timeout":"5000","uuid":"039e269bb0d0bec878ad366eb24b37b1","view_index_dir":"./data"},"compactions":{"_default":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}]","bi_chat":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","bi_client_info":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","bi_data":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","chat":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","client":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]","data":"[{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "23:00"}, {to, "04:00"}]"},"compaction_daemon":{"check_interval":"300","min_file_size":"131072"}}

@janl

This comment has been minimized.

Show comment
Hide comment
@janl

janl Aug 6, 2017

Member

maybe it’s the disk space monitor tripping up on volume paths with spaces in them? Do you get the error when the TM volume is not mounted?

Member

janl commented Aug 6, 2017

maybe it’s the disk space monitor tripping up on volume paths with spaces in them? Do you get the error when the TM volume is not mounted?

@obi458

This comment has been minimized.

Show comment
Hide comment
@obi458

obi458 Aug 7, 2017

  1. TM with spaces, yes: /Volumes/Time Machine Backups/
  2. No error if not mounted, TM mounts every 1 hour, for backups, therefore each 1 hour couchdb goes down

obi458 commented Aug 7, 2017

  1. TM with spaces, yes: /Volumes/Time Machine Backups/
  2. No error if not mounted, TM mounts every 1 hour, for backups, therefore each 1 hour couchdb goes down
@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Aug 10, 2017

Member

For those watching this issue, we have 2 workarounds right now:

  1. Disable Time Machine on your Mac
  2. Disable the compaction daemon in CouchDB

We'll be looking at this issue in closer detail in the coming days.

Member

wohali commented Aug 10, 2017

For those watching this issue, we have 2 workarounds right now:

  1. Disable Time Machine on your Mac
  2. Disable the compaction daemon in CouchDB

We'll be looking at this issue in closer detail in the coming days.

@SCdF

This comment has been minimized.

Show comment
Hide comment
@SCdF

SCdF Aug 31, 2017

I may be having the same issue, as I'm on a Mac and use Time Machine (I also use Arq to backup to the cloud, which covers this directory).

The issue I was having was my CouchDB would randomly start spitting errors over and over that look like this: https://gist.github.com/SCdF/9994ca2afdc727bd39270eaa97abe422

[error] 2017-08-31T08:58:09.717403Z couchdb@localhost emulator -------- Error in process <0.253.0> on node 'couchdb@localhost' with exit value: {{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon... 
[error] 2017-08-31T08:58:09.718157Z couchdb@localhost <0.251.0> -------- gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}}
  last msg: {'EXIT',<0.253.0>,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,273}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]}]}}
     state: {state,<0.253.0>,[]}
[error] 2017-08-31T08:58:09.718506Z couchdb@localhost <0.251.0> -------- CRASH REPORT Process couch_compaction_daemon (<0.251.0>) with 0 neighbors exited with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}} at gen_server:terminate/7(line:804) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_compaction_daemon,init,['Argument__1']}, ancestors: [couch_secondary_services,couch_sup,<0.201.0>], messages: [], links: [<0.210.0>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 2819
[error] 2017-08-31T08:58:09.718956Z couchdb@localhost <0.210.0> -------- Supervisor couch_secondary_services had child compaction_daemon started with couch_compaction_daemon:start_link() at <0.251.0> exit with reason {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}} in context child_terminated

The only way I could find to fix it was basically delete all shard data and re-configure CouchDB.

SCdF commented Aug 31, 2017

I may be having the same issue, as I'm on a Mac and use Time Machine (I also use Arq to backup to the cloud, which covers this directory).

The issue I was having was my CouchDB would randomly start spitting errors over and over that look like this: https://gist.github.com/SCdF/9994ca2afdc727bd39270eaa97abe422

[error] 2017-08-31T08:58:09.717403Z couchdb@localhost emulator -------- Error in process <0.253.0> on node 'couchdb@localhost' with exit value: {{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon... 
[error] 2017-08-31T08:58:09.718157Z couchdb@localhost <0.251.0> -------- gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}}
  last msg: {'EXIT',<0.253.0>,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,273}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]}]}}
     state: {state,<0.253.0>,[]}
[error] 2017-08-31T08:58:09.718506Z couchdb@localhost <0.251.0> -------- CRASH REPORT Process couch_compaction_daemon (<0.251.0>) with 0 neighbors exited with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}} at gen_server:terminate/7(line:804) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_compaction_daemon,init,['Argument__1']}, ancestors: [couch_secondary_services,couch_sup,<0.201.0>], messages: [], links: [<0.210.0>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 2819
[error] 2017-08-31T08:58:09.718956Z couchdb@localhost <0.210.0> -------- Supervisor couch_secondary_services had child compaction_daemon started with couch_compaction_daemon:start_link() at <0.251.0> exit with reason {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}} in context child_terminated

The only way I could find to fix it was basically delete all shard data and re-configure CouchDB.

@SCdF

This comment has been minimized.

Show comment
Hide comment
@SCdF

SCdF Aug 31, 2017

Though if we think it's because of devices mounted with spaces in them, then while I don't have that problem with Time Machine, I do have the Recovery HD mounted by default for some reason:

$ ll /Volumes/
total 8
lrwxr-xr-x   1 root  wheel    1 24 Aug 14:01 Burrich -> /
drwxrwxr-x+ 22 root  wheel  816 29 Aug 18:07 External
drwxrwxrwx   0 root  wheel    0 31 Aug 14:56 MobileBackups
drwxr-xr-x   9 root  wheel  374 24 Aug 14:00 Recovery HD
drwxrwxr-x  14 root  wheel  544  1 Jul 19:36 TimeMachine

SCdF commented Aug 31, 2017

Though if we think it's because of devices mounted with spaces in them, then while I don't have that problem with Time Machine, I do have the Recovery HD mounted by default for some reason:

$ ll /Volumes/
total 8
lrwxr-xr-x   1 root  wheel    1 24 Aug 14:01 Burrich -> /
drwxrwxr-x+ 22 root  wheel  816 29 Aug 18:07 External
drwxrwxrwx   0 root  wheel    0 31 Aug 14:56 MobileBackups
drwxr-xr-x   9 root  wheel  374 24 Aug 14:00 Recovery HD
drwxrwxr-x  14 root  wheel  544  1 Jul 19:36 TimeMachine
@SCdF

This comment has been minimized.

Show comment
Hide comment
@SCdF

SCdF Aug 31, 2017

Ejecting Recovery HD has tentatively fixed the problem for me. I'll report back if that changes.

Is there a reason CouchDB checks for disk space on mounts other than the one it's on?

SCdF commented Aug 31, 2017

Ejecting Recovery HD has tentatively fixed the problem for me. I'll report back if that changes.

Is there a reason CouchDB checks for disk space on mounts other than the one it's on?

@willholley

This comment has been minimized.

Show comment
Hide comment
@willholley

willholley Aug 31, 2017

Member

I ran into a similar issue recently which turned out to be caused by https://bugs.erlang.org/browse/ERL-343.

Member

willholley commented Aug 31, 2017

I ran into a similar issue recently which turned out to be caused by https://bugs.erlang.org/browse/ERL-343.

@cwmichi

This comment has been minimized.

Show comment
Hide comment
@cwmichi

cwmichi Sep 7, 2017

I ran also into this same problem. I downloaded CouchDB 2.1.0 (2017-08-07) for Mac from the CouchDB Website.

My Time Machine is also every 1 hour active for doing the backups ... in this time the CouchDB is not available and crashes.

[error] 2017-09-07T10:08:59.894213Z couchdb@localhost emulator -------- Error in process <0.2586.0> on node 'couchdb@localhost' with exit value: {{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon...
[error] 2017-09-07T10:08:59.894478Z couchdb@localhost <0.2584.0> -------- gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}}
last msg: {'EXIT',<0.2586.0>,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,273}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]}]}}
state: {state,<0.2586.0>,[]}

cwmichi commented Sep 7, 2017

I ran also into this same problem. I downloaded CouchDB 2.1.0 (2017-08-07) for Mac from the CouchDB Website.

My Time Machine is also every 1 hour active for doing the backups ... in this time the CouchDB is not available and crashes.

[error] 2017-09-07T10:08:59.894213Z couchdb@localhost emulator -------- Error in process <0.2586.0> on node 'couchdb@localhost' with exit value: {{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon...
[error] 2017-09-07T10:08:59.894478Z couchdb@localhost <0.2584.0> -------- gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"s..."},...]},...]}}
last msg: {'EXIT',<0.2586.0>,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,526}]},{couch_compaction_daemon,free_space_rec,2,[{file,"src/couch_compaction_daemon.erl"},{line,517}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,273}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]}]}}
state: {state,<0.2586.0>,[]}

@willholley willholley changed the title from MacOS and time machine to CouchDB crashes when on MacOS when a mount point contains whitespace Sep 11, 2017

@willholley willholley changed the title from CouchDB crashes when on MacOS when a mount point contains whitespace to CouchDB crashes when a mount point contains whitespace Sep 11, 2017

@willholley willholley changed the title from CouchDB crashes when a mount point contains whitespace to CouchDB 2.1 crashes when a mount point contains whitespace Sep 11, 2017

willholley added a commit to willholley/couchdb that referenced this issue Sep 11, 2017

Do not crash when free space cannot be calculated
If the compaction daemon cannot calculate the free space
for a volume, do not crash CouchDB. Instead, log a warning
that free space could not be calculated and continue.

Compaction of the database is not necessarily prevented -
just that the disk space for this specific volume
won't be taken into account when deciding whether
to automatically compact or not.

This is primarily to cope with edge cases arising from
ERL-343, whereby disksup:get_disk_data() returns invalid
paths for volumes containing whitespace.

Fixes apache#732

willholley added a commit to willholley/couchdb that referenced this issue Sep 11, 2017

Do not crash when free space cannot be calculated
If the compaction daemon cannot calculate the free space
for a volume, do not crash CouchDB. Instead, log a warning
that free space could not be calculated and continue.

Compaction of the database is not necessarily prevented -
just that the disk space for this specific volume
won't be taken into account when deciding whether
to automatically compact or not.

This is primarily to cope with edge cases arising from
ERL-343, whereby disksup:get_disk_data() returns invalid
paths for volumes containing whitespace.

Fixes apache#732

willholley added a commit to willholley/couchdb that referenced this issue Sep 11, 2017

Do not crash when free space cannot be calculated
If the compaction daemon cannot calculate the free space
for a volume, do not crash CouchDB. Instead, log a warning
that free space could not be calculated and continue.

Compaction of the database is not necessarily prevented -
just that the disk space for this specific volume
won't be taken into account when deciding whether
to automatically compact or not.

This is primarily to cope with edge cases arising from
ERL-343, whereby disksup:get_disk_data() returns invalid
paths for volumes containing whitespace.

Fixes apache#732

willholley added a commit to willholley/couchdb that referenced this issue Sep 12, 2017

Do not crash when free space cannot be calculated
If the compaction daemon cannot calculate the free space
for a volume, do not crash CouchDB. Instead, log a warning
that free space could not be calculated and continue.

Compaction of the database is not necessarily prevented -
just that the disk space for this specific volume
won't be taken into account when deciding whether
to automatically compact or not.

This is primarily to cope with edge cases arising from
ERL-343, whereby disksup:get_disk_data() returns invalid
paths for volumes containing whitespace.

Fixes apache#732

wohali added a commit that referenced this issue Oct 19, 2017

Do not crash when free space cannot be calculated (#803)
If the compaction daemon cannot calculate the free space
for a volume, do not crash CouchDB. Instead, log a warning
that free space could not be calculated and continue.

Compaction of the database is not necessarily prevented -
just that the disk space for this specific volume
won't be taken into account when deciding whether
to automatically compact or not.

This is primarily to cope with edge cases arising from
ERL-343, whereby disksup:get_disk_data() returns invalid
paths for volumes containing whitespace.

Fixes #732
@kinesias

This comment has been minimized.

Show comment
Hide comment
@kinesias

kinesias Oct 24, 2017

When will this issue be fixed?

kinesias commented Oct 24, 2017

When will this issue be fixed?

@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Oct 24, 2017

Member

@kinesias The issue is fixed, on 11 Sep. There just hasn't been a CouchDB release yet that includes this fix.

I gave you a workaround for now in the other ticket you commented on.

Spamming our issue tracker actually makes the release process take longer. Please be patient.

Member

wohali commented Oct 24, 2017

@kinesias The issue is fixed, on 11 Sep. There just hasn't been a CouchDB release yet that includes this fix.

I gave you a workaround for now in the other ticket you commented on.

Spamming our issue tracker actually makes the release process take longer. Please be patient.

@drnybble

This comment has been minimized.

Show comment
Hide comment
@drnybble

drnybble Nov 1, 2017

Did this issue exist in 2.0.0.1? I am seeing the same problem when running CouchDB 2.0.0.1 in a native Windows Docker container with the data directory mounted as a volume into the container.

drnybble commented Nov 1, 2017

Did this issue exist in 2.0.0.1? I am seeing the same problem when running CouchDB 2.0.0.1 in a native Windows Docker container with the data directory mounted as a volume into the container.

@willholley

This comment has been minimized.

Show comment
Hide comment
@willholley

willholley Nov 1, 2017

Member

@drnybble It's a bug in Erlang 19 (will be fixed in Erlang 21) and any CouchDB version prior to 2.1.1 will be susceptible, I think. The fix is to use Erlang 18 or disable the compaction daemon.

Member

willholley commented Nov 1, 2017

@drnybble It's a bug in Erlang 19 (will be fixed in Erlang 21) and any CouchDB version prior to 2.1.1 will be susceptible, I think. The fix is to use Erlang 18 or disable the compaction daemon.

@drnybble

This comment has been minimized.

Show comment
Hide comment
@drnybble

drnybble Nov 3, 2017

@willholley Hmmm, seems that the Windows build of CouchDB is using Erlang 18, so perhaps this is a different problem.

drnybble commented Nov 3, 2017

@willholley Hmmm, seems that the Windows build of CouchDB is using Erlang 18, so perhaps this is a different problem.

@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Nov 3, 2017

Member

You are probably the only person on the planet brave enough to be running Windows as your host OS and Windows as the guest OS for a CouchDB Docker container. This is an unsupported configuration for us, but if you can work out what's wrong, good luck.

Member

wohali commented Nov 3, 2017

You are probably the only person on the planet brave enough to be running Windows as your host OS and Windows as the guest OS for a CouchDB Docker container. This is an unsupported configuration for us, but if you can work out what's wrong, good luck.

@drnybble

This comment has been minimized.

Show comment
Hide comment
@drnybble

drnybble Nov 8, 2017

Sadly updating to 2.1.1 did not fix the issue on Windows.

[error] 2017-11-08T20:48:53.087000Z couchdb@localhost <0.31522.0> -------- gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compacti
on_daemon.erl"},{line,538}]},{couch_compaction_daemon,free_space,1,[{file,"src/couch_compaction_daemon.erl"},{line,512}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,ma
ybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/c..."},...]},...]}}
  last msg: {'EXIT',<0.31524.0>,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,538}]},{couch_compaction_daemon,free_space,1,[{file,"src/couch_compaction_daemon.erl"},{line,512}]},{c
ouch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,
[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.er
l"},{line,195}]}]}}
     state: {state,<0.31524.0>,[]}

drnybble commented Nov 8, 2017

Sadly updating to 2.1.1 did not fix the issue on Windows.

[error] 2017-11-08T20:48:53.087000Z couchdb@localhost <0.31522.0> -------- gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compacti
on_daemon.erl"},{line,538}]},{couch_compaction_daemon,free_space,1,[{file,"src/couch_compaction_daemon.erl"},{line,512}]},{couch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,ma
ybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/c..."},...]},...]}}
  last msg: {'EXIT',<0.31524.0>,{{badmatch,{error,enoent}},[{couch_compaction_daemon,abs_path,1,[{file,"src/couch_compaction_daemon.erl"},{line,538}]},{couch_compaction_daemon,free_space,1,[{file,"src/couch_compaction_daemon.erl"},{line,512}]},{c
ouch_compaction_daemon,can_db_compact,2,[{file,"src/couch_compaction_daemon.erl"},{line,327}]},{couch_compaction_daemon,maybe_compact_db,3,[{file,"src/couch_compaction_daemon.erl"},{line,164}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,
[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.er
l"},{line,195}]}]}}
     state: {state,<0.31524.0>,[]}
@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Nov 8, 2017

Member

As stated in the Windows installer, you must install CouchDB to a path with no spaces and use no spaces in your data directory. We don't have any workaround for this problem at this time.

Member

wohali commented Nov 8, 2017

As stated in the Windows installer, you must install CouchDB to a path with no spaces and use no spaces in your data directory. We don't have any workaround for this problem at this time.

@drnybble

This comment has been minimized.

Show comment
Hide comment
@drnybble

drnybble Nov 9, 2017

I don't believe this problem is related to spaces with paths -- I don't see any spaces.
Unfortunately I don't see anything that stands out as an error in Process Monitor when the error occurs.
I attached a sample of typical output captured by Process Monitor.

erl.txt

drnybble commented Nov 9, 2017

I don't believe this problem is related to spaces with paths -- I don't see any spaces.
Unfortunately I don't see anything that stands out as an error in Process Monitor when the error occurs.
I attached a sample of typical output captured by Process Monitor.

erl.txt

@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Nov 9, 2017

Member

@drnybble Please open a new issue instead of hijacking this one.

Member

wohali commented Nov 9, 2017

@drnybble Please open a new issue instead of hijacking this one.

@apache apache locked and limited conversation to collaborators Nov 9, 2017

willholley added a commit to willholley/couchdb that referenced this issue May 22, 2018

Do not crash when free space cannot be calculated (apache#803)
If the compaction daemon cannot calculate the free space
for a volume, do not crash CouchDB. Instead, log a warning
that free space could not be calculated and continue.

Compaction of the database is not necessarily prevented -
just that the disk space for this specific volume
won't be taken into account when deciding whether
to automatically compact or not.

This is primarily to cope with edge cases arising from
ERL-343, whereby disksup:get_disk_data() returns invalid
paths for volumes containing whitespace.

Fixes apache#732
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.