Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blacklist multiple Erlang releases #1857

Closed
wohali opened this issue Jan 14, 2019 · 10 comments

Comments

@wohali
Copy link
Member

commented Jan 14, 2019

@iilyak ever seen this one?

test/javascript/tests/changes.js                               pass
test/javascript/tests/coffee.js                                pass
test/javascript/tests/compact.js                               pass
test/javascript/tests/config.js                                Error: 
FAIL isRunning
Stacktrace:
        waitForSuccess((function () {CouchDB.last_req = CouchDB.request("GET", "/");return typeof CouchDB.last_req == "object";}),"isRunning")@test/javascript/test_setup.js:64
        @test/javascript/cli_runner.js:46
Failed to execute script.
fail
=======================================================
JavaScript tests complete.
  Failed: 1.  Skipped or passed: 14.
Makefile:231: recipe for target 'javascript' failed
make: *** [javascript] Error 1

It's repeatable at will on my new laptop. In other words, make javascript always fails.

Your Environment

  • Version used: master
  • Operating System and version (desktop or mobile): debian stretch, up to date

@wohali wohali added the testsuite label Jan 14, 2019

@wohali

This comment has been minimized.

Copy link
Member Author

commented Jan 16, 2019

[notice] 2019-01-14T07:56:41.928596Z node1@127.0.0.1 <0.15144.0> -------- rexi_buffer : started servers 
[notice] 2019-01-14T07:56:41.937638Z node1@127.0.0.1 <0.14937.0> -------- config: [features] scheduler set to true for reason nil
[notice] 2019-01-14T07:56:42.007170Z node1@127.0.0.1 <0.15182.0> 640d3f245f 127.0.0.1:15984 127.0.0.1 undefined GET /_node/node1@127.0.0.1/_system 200 ok 1
[notice] 2019-01-14T07:56:42.011893Z node1@127.0.0.1 <0.15182.0> be5192f0d9 127.0.0.1:15984 127.0.0.1 undefined GET /test_suite_db_eywat/bin_doc/foo.txt 200 ok 4
[notice] 2019-01-14T07:56:42.014992Z node1@127.0.0.1 <0.15182.0> d50ba1cf01 127.0.0.1:15984 127.0.0.1 undefined GET /test_suite_db_eywat/ 200 ok 3
[notice] 2019-01-14T07:56:42.015681Z node1@127.0.0.1 <0.15182.0> 18014b7636 127.0.0.1:15984 127.0.0.1 undefined GET /test_suite_db_eywat/ 200 ok 1
[notice] 2019-01-14T07:56:42.016378Z node1@127.0.0.1 <0.15182.0> 964c476f03 127.0.0.1:15984 127.0.0.1 undefined GET /test_suite_db_eywat/ 200 ok 1
[notice] 2019-01-14T07:56:42.017148Z node1@127.0.0.1 <0.15182.0> a6b12270da 127.0.0.1:15984 127.0.0.1 undefined GET /test_suite_db_eywat/ 200 ok 1
nifs/common/prim_file_nif.c:388:owner_death_callback(): Internal error: Failed to defer prim_file close.^M
[notice] 2019-01-14T07:56:42.020951Z node1@127.0.0.1 <0.15182.0> bb28d3dd56 127.0.0.1:15984 127.0.0.1 undefined DELETE /test_suite_db_eywat/?sync=true 200 ok 4
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed^M
[os_mon] memory supervisor port (memsup): Erlang has closed^M

Hmm.

ii  esl-erlang                            1:21.2.2-1                                   amd64        Erlang

Oops, let me try an older Erlang, that's not what's supposed to be installed.

/cc @dch

@wohali

This comment has been minimized.

Copy link
Member Author

commented Jan 16, 2019

OK, that fixed it. We know 21.2 is bad. We should blacklist it in our config setup.

@wohali wohali changed the title [tests] config.js - FAIL isRunning Multiple failures in Erlang 21.2 - should blacklist this release Jan 16, 2019

@wohali

This comment has been minimized.

Copy link
Member Author

commented Jan 16, 2019

Duplicate of #1837.

We might also want to blacklist 20.3.8.10 and lower in the 20 series:

@wohali

This comment has been minimized.

Copy link
Member Author

commented Jan 16, 2019

Well, poop. perhaps we can't.

https://github.com/rebar/rebar/blob/b6d309417c502ca243f810e5313bea36951ef038/src/rebar_require_vsn.erl#L92-L102

Eshell V9.3.3.7  (abort with ^G)
1> erlang:system_info(otp_release).
"20"

That's not enough info to go on. We'd have to write something.

@wohali

This comment has been minimized.

Copy link
Member Author

commented Jan 16, 2019

Looks like rebar3 gives us blacklisted_otp_versions which is what we really want here.

erlang/rebar3@04fedc7

So this is really dependent on #1428 .

@AlexanderKaraberov

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2019

@wohali Does this mean that it's not safe to update our production environment to OTP 21.2? (actually we already did this). We are using the latest CouchDB 2.3+our patch set. It's been several days since we migrated to OTP 21.2 and haven't observed any serious deterioration (as of yet). What would be your or other core maintainers advice? Shall we rollback to OTP 21.1 to be sure all the things work?

@nickva

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2019

There is another one as well:

https://bugs.erlang.org/browse/ERL-830

This one might affect systems which use TLS connections (either as a server or via ibrowse as replication clients). This seems to be fixed in 21.2.3 (just released).

@nickva

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2019

@AlexanderKaraberov

https://bugs.erlang.org/browse/ERL-821 : would affect tests so running tests would not work but would not affect production. This is fixed in the latest 21.2.3

https://bugs.erlang.org/browse/ERL-827 : not sure about this one, it is manifesting on FreeBSD, perhaps that's a necessary trigger for the bug and on other machines (Linux) might not happen. This is not fixed yet and and the cause it not known yet.

https://bugs.erlang.org/browse/ERL-830 : not sure how likely this one is to happen, it does seem serious. One the other hand, I'd imagine if random bytes of data are not sent it would break http protocol and json formatting so connections would be retried. This is also fixed in 21.2.3

Would you have the option of building a point release? Then I would try 21.2.3 perhaps. To be even more stable perhaps try 20.3.8.14, I've seen that run in production across hundreds of machine already for months without any issues.

@AlexanderKaraberov

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2019

Hi @nickva ,
Thank you for a prompt and detailed feedback.

Would you have the option of building a point release? Then I would try 21.2.3 perhaps.

This option is good for us. And from what I see it addresses some of the pinpointed problems. OTP 21.x delivers a lot of performance improvements to I/O subsystem hence we would be happy to stay on this major release instead of rolling back to 20.x

@wohali

This comment has been minimized.

Copy link
Member Author

commented Jan 21, 2019

We just heard from @dch that 21.2.3 seems to have fixed ERL-827. So it will be sufficient to blacklist 21.2, 21.2.1, and 21.2.2.

@wohali wohali changed the title Multiple failures in Erlang 21.2 - should blacklist this release Blacklist Erlang 21.2, 21.2.1, 21.2.2 (21.2.3+ OK) Jan 21, 2019

wohali added a commit that referenced this issue Jan 22, 2019
wohali added a commit that referenced this issue Jan 22, 2019
wohali added a commit that referenced this issue Jan 22, 2019
wohali added a commit that referenced this issue Jan 22, 2019
wohali added a commit that referenced this issue Jan 22, 2019
wohali added a commit that referenced this issue Jan 22, 2019
wohali added a commit that referenced this issue Jan 23, 2019
wohali added a commit that referenced this issue Jan 23, 2019

@wohali wohali closed this in #1871 Jan 23, 2019

wohali added a commit that referenced this issue Jan 23, 2019

@wohali wohali changed the title Blacklist Erlang 21.2, 21.2.1, 21.2.2 (21.2.3+ OK) Blacklist multiple Erlang releases Jun 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.