New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RHEL / CentOS 7 `js-1.8.5-20` package breaks couchjs #1293

Closed
wohali opened this Issue Apr 26, 2018 · 14 comments

Comments

Projects
None yet
5 participants
@wohali
Copy link
Member

wohali commented Apr 26, 2018

I have received 2 reports by email that this is failing. One follows below.


Hello,

I've noticed your message in couchdb-usr mailing list, unfortunately didn't figure out how to get old messages in my mailbox so can reply.

We had to rollback for our server to js-1.8.5-19 and server works fine.

yum changelog all js returns following

====================AvailablePackages====================
1:js-1.8.5-20.el7.i686nfs_rhel_updates
*ThuApr614:00:002017YaakovSelkowitz<yselkowi@redhat.com>-1:1.8.5-20
-Fixfor48-bitVAonaarch64
-Resolves:#1423015

*TueAug2614:00:002014YaakovSelkowitz<yselkowi@redhat.com>-1:1.8.5-19
-Rebaseaarch64patch
-Resolves:#1134124

The error in couchdb is

[info]2018-04-26T12:19:44.675788Zcouchdb@127.0.0.1<0.213.0>--------couch_proc_manager<0.2783.0>diednormal
[error]2018-04-26T12:19:44.675886Zcouchdb@127.0.0.1<0.2781.0>--------OSProcessError<0.2783.0>::{parse_error,bad_token}

Couchdb seems to work fine if disabling javascript query server

I've did a lookup on CentOS GIT repos and found following
https://git.centos.org/commit/rpms!js.git/4eb227977009db1c3df47a2283a85b3150ea223f

Hope this helps


The workaround is to temporarily revert to the js-1.8.5-19 package.

@wohali wohali added the packaging label Apr 26, 2018

@wohali

This comment has been minimized.

Copy link
Member

wohali commented Apr 27, 2018

I have filed https://bugs.centos.org/view.php?id=14720 to try and track this issue. We'll see if anything gets done about it.

@wohali

This comment has been minimized.

Copy link
Member

wohali commented Apr 27, 2018

After a full clean build and ensuring no traces of older packages or hand-compiled libmozjs are present on a CentOS 7 box, I am still getting a failure in our JS test suite with the upstream package, but now it's in users_db_security.js specifically - couchjs does run correctly most of the time.

The failure is:

test/javascript/tests/users_db_security.js
    Error: Failed to execute HTTP request: Failed connect to 127.0.0.1:15984; Connection refused
Trace back (most recent call first):

  37: test/javascript/couch_http.js
      ("\"10\"\n")
 468: 127.0.0.1/_config/couch_httpd_auth/iterations",[object Object])@test/javascript/couch.js
      ("PUT","/_node/node1
 408: test/javascript/couch_test_runner.js
      run_on_modified_server([object Array],(function () {try {testFun(schem
 416: test/javascript/tests/users_db_security.js
      ("bcrypt",1,[object Array])
 404: test/javascript/tests/users_db_security.js
      ()
  37: test/javascript/cli_runner.js
      runTest()
  48: test/javascript/cli_runner.js

fail
@wohali

This comment has been minimized.

Copy link
Member

wohali commented Apr 27, 2018

The error in couch.log is:

enif_send: env==NULL on non-SMP VM

with a CouchDB crash.

Looks like this is a known bug on single processor machines and is fixable.

@wohali

This comment has been minimized.

Copy link
Member

wohali commented Apr 27, 2018

By sticking -smp enable in rel/overlay/etc/vm.args as a temporary workaround, I'm able to pass all JS tests using upstream's js-1.8.5-20 package. So the problem resides in our couchdb-ci repo and its use of a locally compiled js that is incompatible with the version provided in the package.

I will come back to this next week to resolve.

bbrowning added a commit to projectodd/openwhisk-openshift that referenced this issue May 15, 2018

Temporarily pin js dep for CouchDB until we work out a proper fix
CentOS 7 bumped its `js` version from 1.8.5-19 to 1.8.5-20 and
apparently broke CouchDB's JavaScript view generation in the
process. apache/couchdb#1293 tracks that
issue.

So, for now, we pin to 1.8.5-19. It's not available to install
directly from the repos so we point to a download location on
vault.centos.org.
@thescrublet

This comment has been minimized.

Copy link

thescrublet commented May 17, 2018

Just in case you didn't have enough information on this already, when we found this occurring we tracked it down to ONLY PUT requests with our design documents:

couchdb_1 | SyntaxError: JSON.parse
couchdb_1 | SyntaxError: JSON.parse
couchdb_1 | Stacktrace:
couchdb_1 | ()@./share/server/main.js:1576
couchdb_1 | ()@./share/server/main.js:1603
couchdb_1 | @./share/server/main.js:1
couchdb_1 | Failed to execute script.
couchdb_1 | Stacktrace:
couchdb_1 | ()@./share/server/main.js:1576
couchdb_1 | ()@./share/server/main.js:1603
couchdb_1 | @./share/server/main.js:1
couchdb_1 | Failed to execute script.

And that couchdb log would scroll until the service was stopped. We used the following to trigger the error:
curl -vX PUT -H “Content-Type: application/json” http://127.0.0.1:5984/our-database/_design/dom -d ‘{ “_id”: “_design/dom”, “views”: { “dom”: { “map”: “function (doc) { // our function code }” } }, “language”: “javascript” }’

PUTs of non-design-docs worked, POSTs and GETs worked, it was just PUT of /_design/ddoc that was a problem.

@wohali

This comment has been minimized.

Copy link
Member

wohali commented May 17, 2018

Thanks @thescrublet , this is consistent with our understanding.

We're going to work around this by shipping our own js packages that replace the os-provided ones on all binary platforms.

We hope to have a new binary release soon.

@cbonnissent

This comment has been minimized.

Copy link

cbonnissent commented May 23, 2018

Hello,

I have the same issue with user creation. The first creation is OK and the following make an error 500 with this stack trace :

[error] 2018-05-22T09:07:07.613386Z couchdb@127.0.0.1 <0.17418.240> -------- rexi_server: from:   couchdb@127.0.0.1(<0.15783.240>) mfa: fabric_rpc:update_docs/3 exit:{timeout,{gen_server,call,  [couch_proc_manager,{get_proc,{doc,<<"_design/_auth">>, {1,[<<199,155,192,12,136,156,233,185,18,251,222,138,63,82,222,55>>]},{[{<<"language">>, <<"javascript">>},{<<"validate_doc_update">>,<<"
function(newDoc, oldDoc, userCtx, secObj) {
    if (newDoc._deleted === true) {
        // allow deletes by admins and matching users
        // without checking the other fields
        if ((userCtx.roles.indexOf('_admin') !== -1) ||
            (userCtx.name == oldDoc.name)) {
            return;
        } else {
            throw({forbidden: 'Only admins may delete other user docs.'});
        }
    }

    if (newDoc.type !== 'user') {
        throw({forbidden : 'doc.type must be user'});
    } // we only allow user docs for now

    if (!newDoc.name) {
        throw({forbidden: 'doc.name is required'});
    }

    if (!newDoc.roles) {
        throw({forbidden: 'doc.roles must exist'});
    }

    if (!isArray(newDoc.roles)) {
        throw({forbidden: 'doc.roles must be an array'});
    }

    for (var idx = 0; idx < newDoc.roles.length; idx++) {
        if (typeof newDoc.roles[idx] !== 'string') {
            throw({forbidden: 'doc.roles can only contain strings'});
        }
    }

    if (newDoc._id !== ('org.couchdb.user:' + newDoc.name)) {
        throw({
            forbidden: 'Doc ID must be of the form org.couchdb.user:name'
        });
    }

    if (oldDoc) { // validate all updates
        if (oldDoc.name !== newDoc.name) {
            throw({forbidden: 'Usernames can not be changed.'});
        }
    }

    if (newDoc.password_sha && !newDoc.salt) {
        throw({
            forbidden: 'Users with password_sha must have a salt.' +
                'See /_utils/script/couch.js for example code.'
        });
    }

    if (newDoc.password_scheme === \"pbkdf2\") {
        if (typeof(newDoc.iterations) !== \"number\") {
           throw({forbidden: \"iterations must be a number.\"});
        }
        if (typeof(newDoc.derived_key) !== \"string\") {
           throw({forbidden: \"derived_key must be a string.\"});
        }
    }

    var is_server_or_database_admin = function(userCtx, secObj) {
        // see if the user is a server admin
        if(userCtx.roles.indexOf('_admin') !== -1) {
            return true; // a server admin
        }

        // see if the user a database admin specified by name
        if(secObj && secObj.admins && secObj.admins.names) {
            if(secObj.admins.names.indexOf(userCtx.name) !== -1) {
                return true; // database admin
            }
        }

        // see if the user a database admin specified by role
        if(secObj && secObj.admins && secObj.admins.roles) {
            var db_roles = secObj.admins.roles;
            for(var idx = 0; idx < userCtx.roles.length; idx++) {
                var user_role = userCtx.roles[idx];
                if(db_roles.indexOf(user_role) !== -1) {
                    return true; // role matches!
                }
            }
        }

        return false; // default to no admin
    }

    if (!is_server_or_database_admin(userCtx, secObj)) {
        if (oldDoc) { // validate non-admin updates
            if (userCtx.name !== newDoc.name) {
                throw({
                    forbidden: 'You may only update your own user document.'
                });
            }
            // validate role updates
            var oldRoles = (oldDoc.roles || []).sort();
            var newRoles = newDoc.roles.sort();

            if (oldRoles.length !== newRoles.length) {
                throw({forbidden: 'Only _admin may edit roles'});
            }

            for (var i = 0; i < oldRoles.length; i++) {
                if (oldRoles[i] !== newRoles[i]) {
                    throw({forbidden: 'Only _admin may edit roles'});
                }
            }
        } else if (newDoc.roles.length > 0) {
            throw({forbidden: 'Only _admin may set roles'});
        }
    }

    // no system roles in users db
    for (var i = 0; i < newDoc.roles.length; i++) {
        if (newDoc.roles[i][0] === '_') {
            throw({
                forbidden:
                'No system roles (starting with underscore) in users db.'
            });
        }
    }

    // no system names as names
    if (newDoc.name[0] === '_') {
        throw({forbidden: 'Username may not start with underscore.'});
    }

    var badUserNameChars = [':'];

    for (var i = 0; i < badUserNameChars.length; i++) {
        if (newDoc.name.indexOf(badUserNameChars[i]) >= 0) {
            throw({forbidden: 'Character `' + badUserNameChars[i] +
                    '` is not allowed in usernames.'});
        }
    }
}
">>}]},[],false,[]},{<<"_design/_auth">>,<<"1-c79bc00c889ce9b912fbde8a3f52de37">>}},5000]}} [{gen_server,call,3,[{file,"gen_server.erl"},{line,212}]},{couch_query_servers,get_ddoc_process,2,[{file,"src/couch_query_servers.erl"},{line,498}]},{couch_query_servers,with_ddoc_proc,2,[{file,"src/couch_query_servers.erl"},{line,463}]},{couch_query_servers,validate_doc_update,5,[{file,"src/couch_query_servers.erl"},{line,318}]},{couch_db,'-validate_doc_update_int/3-lc$^0/1-0-',5,[{file,"src/couch_db.erl"},{line,784}]},{couch_db,'-validate_doc_update_int/3-fun-0-',3,[{file,"src/couch_db.erl"},{line,784}]},{couch_stats,update_histogram,2,[{file,"src/couch_stats.erl"},{line,102}]},{couch_db,'-prep_and_validate_updates/6-fun-1-',3,[{file,"src/couch_db.erl"},{line,878}]}]

``

I have reverted the js package to the version 1.8.5-19 and it's work

@wohali

This comment has been minimized.

Copy link
Member

wohali commented May 28, 2018

As early advice to anyone bitten by this bug, once you have the CouchDB repo added to your install, you can just:

yum install -y couch-js

to get a working library that replaces the broken one from CentOS.

The very next push of CouchDB builds to the repo will require this new couch-js package instead of the js package. These builds are done - and are undergoing upgrade testing right now. They will be released shortly.

Packages have been temporarily removed due to conflicts with the js package and the old CouchDB .rpms. Once I have finished testing the new CouchDB .rpms I will re-publish these packages. Sorry for the inconvenience.

@PlaidShirt

This comment has been minimized.

Copy link

PlaidShirt commented Jun 5, 2018

I seem to have run into this problem as well. Are there any workarounds for it? I've looked at downgrading my js install but that doesn't seem to be a viable option on this particular machine.

@wohali

This comment has been minimized.

Copy link
Member

wohali commented Jun 5, 2018

Pin to version 19 of the js library, or wait for our updated packages that remove the dependency on the OS provided js-1.8.5 package.

@PlaidShirt

This comment has been minimized.

Copy link

PlaidShirt commented Jun 5, 2018

Thanks for the fast response! I was having troubles with getting the package downgraded, but after a full night sleep and caffeine I'm up and running with it pinned. I'll watch this space for updates.

@wohali

This comment has been minimized.

Copy link
Member

wohali commented Jun 14, 2018

The new packages are ready and will be uploaded soon - we are performing final acceptance testing on them.

@wasperen

This comment has been minimized.

Copy link

wasperen commented Jun 19, 2018

New install; cannot get it off the ground... I found an old copy of js-1.8.5-19 and installed that. Is it worth waiting a little to get the latest couchdb rpm?

@wohali

This comment has been minimized.

Copy link
Member

wohali commented Jul 10, 2018

Packages for the brand new CouchDB 2.1.2 have been published.

If there are any issues with the new packages, please open a new ticket.

@apache apache locked as resolved and limited conversation to collaborators Jul 10, 2018

@wohali wohali closed this Jul 10, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.