Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Elixir 1.2: Logger crash when using `inspect` in production with protocols consolidated #4518
Wasn't sure whether to report this as not certain it's an elixir/logger bug rather than something I'm doing wrong (tried slack and SO first), but on I figure logger crashing seems likely to be a bug that I should report even if I am doing something odd. If it isn't a bug, many apologies.
So: the first
The code that triggers it is just
Elixir version: 1.2.2
We're using the heroku elixir buildpack to make slugs, which are then run using docker.
The weird thing is, this crash only happens in production. The same app running locally -- even when using MIX_ENV=prod, running from the same slug that runs in production -- does not have any problems.
Let me know what other details I can provide to help.
Thanks in advance :)
It is failing because it thinks the function Elixir.Inspect.List.inspect/2 is not available. I would look for the following things:
Hi Jose! Thanks for the quick reply :)
I'm a little confused as to how it can not be, but you're right, it isn't.
Not sure how else to check if it's 'present inside Elixir'? Grepping the
I'm not doing anything that redefines any Elixir builtins, so unless I'm missing something, I'm afraid I don't see how I could be. (And as I said,
This seems to be the root cause. What does
After some experimentation -- it seems to be random :(
Trying the non-boolean version: I obviously can't
To be clear, this isn't just nondeterminism between builds, it's between deploys of a single build. I.e. I can deploy the exact same slug (same SHA) several times, and sometimes ensure_loaded will fail with
That's quite surprising.
When you mention "deploy" the same slug, does it compile the code every time you deploy or you are referring to the exact same compiled artifacts? Would you be able to retrieve the
Thank you for following up.
Sorry for ambiguity - I mean literally the exact same compiled artifacts. heroku-elixir-buildpack spits out a tarball with everything compiled, that's what I'm referring to by the 'slug'. 'deploying the slug' is just starting a docker container with the slug extracted to /app, and running this script (passing in
I've put the beam file from a slug up here in case it helps. Since it does sometimes work, I'm guessing everything is in there that should be else it couldn't ever work. :/
No, thank you for helping! I realise this must be frustrating to debug remotely.
After some more experimentation, turns out this bug only ever happens if the very first call to
In other words, the issue completely disappears if I do something like
(I've also made sure it wasn't an issue with one of my other deps - I can reproduce the issue with a dummy app with no external deps except cowboy (1.0.0), and no other code except an app callback that starts cowboy and one http handler that does an IO.inspect).
The workaround solves the pressing issue for us, but of course I'm happy to keep trying any experiments you want me to to track the underlying bug down.
We think we know why this was happening now, and it was our fault. Not a bug in elixir or erlang. Sorry for the invalid bug report :(
If you're interested:
Since the things in /tmp are overlayfs upper directories, this cleanup script didn't affect the underlying slug, which was still correct. But it did mark files as 'deleted' in overlayfs, so they were hidden from the docker container.
Result: random sadness, depending on a combination of when each .beam file was last modified, and which files had already been loaded by the erlang vm by the time the cleanup script ran. (So I guess my workaround only worked because it meant those files were loaded early on - and presumably most other protocols were either exercised early on or else not at all, so the problem only showed up with
We've now moved the overlayfs upperdirs out of /tmp, and everything is working fine. (Why this hasn't shown up as an issue for us before, who knows..)
Thank you for your help earlier, and apologies again!