OTP-20 Support

[chewbranca]

While doing a fresh CouchDB build on OTP-20 I ran into blocking issues with the removal of the deprecated crypto:rand_bytes/1 function. This function was deprecated in [1] in the OTP-19 release and has now been completely removed in OTP-20 with [2]. You can see in [1] that there is an alternative function crypto:strong_rand_bytes/1 that is now being used in Erlang. I don't know the nuances of that change well enough to conclude whether this would be appropriate for us to use. It does appear that the current incarnation of crypto:strong_rand_bytes/1 has been in since OTP-17 [3], and an earlier version even further.

We currently used crypto:rand_bytes in a number of modules, and even in 3rd party deps like Mochiweb:

$ grep -r crypto:rand_bytes src/
src//b64url/test/benchmark.escript:    Data = crypto:rand_bytes(Size),
src//chttpd/src/chttpd.erl:    Nonce = couch_util:to_hex(crypto:rand_bytes(5)),
src//couch/src/couch_uuids.erl:    list_to_binary(couch_util:to_hex(crypto:rand_bytes(16))).
src//couch/src/couch_uuids.erl:    utc_suffix(couch_util:to_hex(crypto:rand_bytes(9))).
src//couch/src/couch_uuids.erl:    couch_util:to_hex((crypto:rand_bytes(13))).
src//couch/test/couch_work_queue_tests.erl:            Item = crypto:rand_bytes(Size),
src//couch/test/couchdb_views_tests.erl:                {<<"value">>, base64:encode(crypto:rand_bytes(1000))}
src//couch_replicator/test/couch_replicator_large_atts_tests.erl:        {data, fun(Count) -> crypto:rand_bytes(Count) end}
src//couch_replicator/test/couch_replicator_many_leaves_tests.erl:                AttData = crypto:rand_bytes(100),
src//couch_replicator/test/couch_replicator_missing_stubs_tests.erl:    AttData = crypto:rand_bytes(6000),
src//couch_replicator/test/couch_replicator_missing_stubs_tests.erl:                body = {[{<<"value">>, base64:encode(crypto:rand_bytes(100))}]}
src//ddoc_cache/src/ddoc_cache_util.erl:    to_hex(crypto:rand_bytes(16), []).
src//mochiweb/src/mochiweb_multipart.erl:                            mochihex:to_hex(crypto:rand_bytes(8))).
src//mochiweb/src/mochiweb_session.erl:    IV = crypto:rand_bytes(16),
src//mochiweb/src/mochiweb_session.erl:    IV = crypto:rand_bytes(16),
src//mochiweb/test/mochiweb_tests.erl:                    Body = crypto:rand_bytes(Size),
src//oauth/src/oauth.erl:  Nonce = base64:encode_to_string(crypto:rand_bytes(32)), % cf. ruby-oauth
src//snappy/README.md:18&gt; Large = base64:encode(crypto:rand_bytes(100 * 1024)).

In the ticket title I've intentionally left crypto:rand_bytes out as I have not verified this is the only deprecated function preventing CouchDB from building on OTP-20.

[1] erlang/otp@1ad1883
[2] erlang/otp@a302543
[3] erlang/otp@973c10a

Expected Behavior

Expect to be able to boot CouchDB on OTP-20.

Current Behavior

Cannot boot CouchDB on OTP-20.

Possible Solution

Remove all uses of removed deprecated functions.

Steps to Reproduce (for bugs)

1. Build OTP-20
2. Use OTP-20 to compile CouchDB
3. Run CouchDB dev server
4. See runtime errors for undef functions

[chewbranca]

Well on a positive note it appears crypto:rand_bytes is the only undefined function preventing CouchDB from booting. I ran for f in $(grep -rl crypto:rand_bytes src/); do sed -i 's/rand_bytes/strong_rand_bytes/' $f; done and I was able to successfully boot CouchDB and query the welcome handler. I forgot to mention in the initial write up of this ticket that this is a runtime error, not a compile time error, so it's possible there are other now missing functions. I have not yet run the test suite with OTP-20 to look for other problematic areas.

[wohali]

From IRC:

<+rnewson> the rand_bytes deprecation should be handled in couch_crypto following the same pattern

[chewbranca]

RE using couch_crypto, I think that logically makes sense for the couch Erlang app, but anything that does not depend on that app or is a dependency of that app will need a different approach. Similarly, mochiweb and snappy are external dependencies and will need to be handled explicitly.

[ArchangeGabriel]

This was also reported here https://issues.apache.org/jira/browse/COUCHDB-3414, originating from https://bugs.archlinux.org/task/53499.

During build, you can see the following messages:

 /build/couchdb/src/apache-couchdb-2.0.0/src/couch_epi/src/couch_epi_util.erl:33: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2 /build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:25: Warning: call to crypto:sha/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:27: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:37: Warning: call to crypto:sha_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:39: Warning: call to crypto:md5_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:50: Warning: call to crypto:sha_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:52: Warning: call to crypto:md5_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:63: Warning: call to crypto:sha_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:65: Warning: call to crypto:md5_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:76: Warning: call to crypto:sha_mac/2 will fail, since it was removed in 20.0; use crypto:hmac/3

Build does still success though, but on start-up attempt:

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttpd_auth|keeper\",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_util,hash,1,[{file,\"src/couch_epi_util.erl\"},{line,25}]},{couch_epi_functions,data,1,[{file,\"src/couch_epi_functions.erl\"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,\"src/couch_epi_module_keeper.erl\"},{line,116}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"}

A rough patch was proposed here: https://bugs.archlinux.org/task/53499#comment158393

[drasko]

I did a build on OTP20 Debian Testing (Buster) - seems to be working fine. Deprication warnings along the way, but nothing critical IMHO.

[nickva]

It compiles on 20.0 and EUnit tests run.