New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OTP-20 Support #617

Closed
chewbranca opened this Issue Jun 24, 2017 · 6 comments

Comments

Projects
None yet
5 participants
@chewbranca
Contributor

chewbranca commented Jun 24, 2017

While doing a fresh CouchDB build on OTP-20 I ran into blocking issues with the removal of the deprecated crypto:rand_bytes/1 function. This function was deprecated in [1] in the OTP-19 release and has now been completely removed in OTP-20 with [2]. You can see in [1] that there is an alternative function crypto:strong_rand_bytes/1 that is now being used in Erlang. I don't know the nuances of that change well enough to conclude whether this would be appropriate for us to use. It does appear that the current incarnation of crypto:strong_rand_bytes/1 has been in since OTP-17 [3], and an earlier version even further.

We currently used crypto:rand_bytes in a number of modules, and even in 3rd party deps like Mochiweb:

$ grep -r crypto:rand_bytes src/
src//b64url/test/benchmark.escript:    Data = crypto:rand_bytes(Size),
src//chttpd/src/chttpd.erl:    Nonce = couch_util:to_hex(crypto:rand_bytes(5)),
src//couch/src/couch_uuids.erl:    list_to_binary(couch_util:to_hex(crypto:rand_bytes(16))).
src//couch/src/couch_uuids.erl:    utc_suffix(couch_util:to_hex(crypto:rand_bytes(9))).
src//couch/src/couch_uuids.erl:    couch_util:to_hex((crypto:rand_bytes(13))).
src//couch/test/couch_work_queue_tests.erl:            Item = crypto:rand_bytes(Size),
src//couch/test/couchdb_views_tests.erl:                {<<"value">>, base64:encode(crypto:rand_bytes(1000))}
src//couch_replicator/test/couch_replicator_large_atts_tests.erl:        {data, fun(Count) -> crypto:rand_bytes(Count) end}
src//couch_replicator/test/couch_replicator_many_leaves_tests.erl:                AttData = crypto:rand_bytes(100),
src//couch_replicator/test/couch_replicator_missing_stubs_tests.erl:    AttData = crypto:rand_bytes(6000),
src//couch_replicator/test/couch_replicator_missing_stubs_tests.erl:                body = {[{<<"value">>, base64:encode(crypto:rand_bytes(100))}]}
src//ddoc_cache/src/ddoc_cache_util.erl:    to_hex(crypto:rand_bytes(16), []).
src//mochiweb/src/mochiweb_multipart.erl:                            mochihex:to_hex(crypto:rand_bytes(8))).
src//mochiweb/src/mochiweb_session.erl:    IV = crypto:rand_bytes(16),
src//mochiweb/src/mochiweb_session.erl:    IV = crypto:rand_bytes(16),
src//mochiweb/test/mochiweb_tests.erl:                    Body = crypto:rand_bytes(Size),
src//oauth/src/oauth.erl:  Nonce = base64:encode_to_string(crypto:rand_bytes(32)), % cf. ruby-oauth
src//snappy/README.md:18&gt; Large = base64:encode(crypto:rand_bytes(100 * 1024)).

In the ticket title I've intentionally left crypto:rand_bytes out as I have not verified this is the only deprecated function preventing CouchDB from building on OTP-20.

[1] erlang/otp@1ad1883
[2] erlang/otp@a302543
[3] erlang/otp@973c10a

Expected Behavior

Expect to be able to boot CouchDB on OTP-20.

Current Behavior

Cannot boot CouchDB on OTP-20.

Possible Solution

Remove all uses of removed deprecated functions.

Steps to Reproduce (for bugs)

  1. Build OTP-20
  2. Use OTP-20 to compile CouchDB
  3. Run CouchDB dev server
  4. See runtime errors for undef functions
@chewbranca

This comment has been minimized.

Show comment
Hide comment
@chewbranca

chewbranca Jun 24, 2017

Contributor

Well on a positive note it appears crypto:rand_bytes is the only undefined function preventing CouchDB from booting. I ran for f in $(grep -rl crypto:rand_bytes src/); do sed -i 's/rand_bytes/strong_rand_bytes/' $f; done and I was able to successfully boot CouchDB and query the welcome handler. I forgot to mention in the initial write up of this ticket that this is a runtime error, not a compile time error, so it's possible there are other now missing functions. I have not yet run the test suite with OTP-20 to look for other problematic areas.

Contributor

chewbranca commented Jun 24, 2017

Well on a positive note it appears crypto:rand_bytes is the only undefined function preventing CouchDB from booting. I ran for f in $(grep -rl crypto:rand_bytes src/); do sed -i 's/rand_bytes/strong_rand_bytes/' $f; done and I was able to successfully boot CouchDB and query the welcome handler. I forgot to mention in the initial write up of this ticket that this is a runtime error, not a compile time error, so it's possible there are other now missing functions. I have not yet run the test suite with OTP-20 to look for other problematic areas.

@chewbranca chewbranca changed the title from CouchDB won't build on OTP-20 to CouchDB won't boot on OTP-20 Jun 24, 2017

@wohali

This comment has been minimized.

Show comment
Hide comment
@wohali

wohali Jun 24, 2017

Member

From IRC:

<+rnewson> the rand_bytes deprecation should be handled in couch_crypto following the same pattern
Member

wohali commented Jun 24, 2017

From IRC:

<+rnewson> the rand_bytes deprecation should be handled in couch_crypto following the same pattern

@wohali wohali added the build label Jun 24, 2017

@chewbranca

This comment has been minimized.

Show comment
Hide comment
@chewbranca

chewbranca Jun 24, 2017

Contributor

RE using couch_crypto, I think that logically makes sense for the couch Erlang app, but anything that does not depend on that app or is a dependency of that app will need a different approach. Similarly, mochiweb and snappy are external dependencies and will need to be handled explicitly.

Contributor

chewbranca commented Jun 24, 2017

RE using couch_crypto, I think that logically makes sense for the couch Erlang app, but anything that does not depend on that app or is a dependency of that app will need a different approach. Similarly, mochiweb and snappy are external dependencies and will need to be handled explicitly.

@ArchangeGabriel

This comment has been minimized.

Show comment
Hide comment
@ArchangeGabriel

ArchangeGabriel Jul 21, 2017

This was also reported here https://issues.apache.org/jira/browse/COUCHDB-3414, originating from https://bugs.archlinux.org/task/53499.

During build, you can see the following messages:

 /build/couchdb/src/apache-couchdb-2.0.0/src/couch_epi/src/couch_epi_util.erl:33: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2 /build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:25: Warning: call to crypto:sha/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:27: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:37: Warning: call to crypto:sha_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:39: Warning: call to crypto:md5_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:50: Warning: call to crypto:sha_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:52: Warning: call to crypto:md5_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:63: Warning: call to crypto:sha_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:65: Warning: call to crypto:md5_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:76: Warning: call to crypto:sha_mac/2 will fail, since it was removed in 20.0; use crypto:hmac/3

Build does still success though, but on start-up attempt:

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttpd_auth|keeper\",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_util,hash,1,[{file,\"src/couch_epi_util.erl\"},{line,25}]},{couch_epi_functions,data,1,[{file,\"src/couch_epi_functions.erl\"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,\"src/couch_epi_module_keeper.erl\"},{line,116}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"}

A rough patch was proposed here: https://bugs.archlinux.org/task/53499#comment158393

ArchangeGabriel commented Jul 21, 2017

This was also reported here https://issues.apache.org/jira/browse/COUCHDB-3414, originating from https://bugs.archlinux.org/task/53499.

During build, you can see the following messages:

 /build/couchdb/src/apache-couchdb-2.0.0/src/couch_epi/src/couch_epi_util.erl:33: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2 /build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:25: Warning: call to crypto:sha/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:27: Warning: call to crypto:md5/1 will fail, since it was removed in 20.0; use crypto:hash/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:37: Warning: call to crypto:sha_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:39: Warning: call to crypto:md5_init/0 will fail, since it was removed in 20.0; use crypto:hash_init/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:50: Warning: call to crypto:sha_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:52: Warning: call to crypto:md5_update/2 will fail, since it was removed in 20.0; use crypto:hash_update/2
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:63: Warning: call to crypto:sha_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:65: Warning: call to crypto:md5_final/1 will fail, since it was removed in 20.0; use crypto:hash_final/1
/build/couchdb/src/apache-couchdb-2.0.0/src/couch/src/couch_crypto.erl:76: Warning: call to crypto:sha_mac/2 will fail, since it was removed in 20.0; use crypto:hmac/3

Build does still success though, but on start-up attempt:

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,couch_epi,{{shutdown,{failed_to_start_child,\"couch_epi|chttpd_auth|keeper\",{undef,[{crypto,md5,[<<131,106>>],[]},{couch_epi_util,hash,1,[{file,\"src/couch_epi_util.erl\"},{line,25}]},{couch_epi_functions,data,1,[{file,\"src/couch_epi_functions.erl\"},{line,33}]},{couch_epi_module_keeper,do_reload_if_updated,1,[{file,\"src/couch_epi_module_keeper.erl\"},{line,116}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}},{couch_epi_app,start,[normal,[]]}}}"}

A rough patch was proposed here: https://bugs.archlinux.org/task/53499#comment158393

@drasko

This comment has been minimized.

Show comment
Hide comment
@drasko

drasko Sep 14, 2017

I did a build on OTP20 Debian Testing (Buster) - seems to be working fine. Deprication warnings along the way, but nothing critical IMHO.

drasko commented Sep 14, 2017

I did a build on OTP20 Debian Testing (Buster) - seems to be working fine. Deprication warnings along the way, but nothing critical IMHO.

@wohali wohali changed the title from CouchDB won't boot on OTP-20 to OTP-20 Support Oct 1, 2017

@nickva

This comment has been minimized.

Show comment
Hide comment
@nickva

nickva Oct 16, 2017

Contributor

It compiles on 20.0 and EUnit tests run.

Contributor

nickva commented Oct 16, 2017

It compiles on 20.0 and EUnit tests run.

@nickva nickva closed this Oct 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment