Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ping command is crashing because of race conditions #645

Open
AndrewDryga opened this Issue Mar 10, 2019 · 0 comments

Comments

Projects
None yet
1 participant
@AndrewDryga
Copy link
Contributor

AndrewDryga commented Mar 10, 2019

We are using ping command as a Kubernetes liveness probe and it shows itself as extremely not stable:

  1. Because it uses static names it would collide when intervals are too frequent:
{"log":"Protocol 'inet_tcp': the name talkinto-domain_maint_@10-16-0-4.default.pod.cluster.local seems to be in use by another Erlang node\n","logging.googleapis.com/sourceLocation":{"file":null,"line":0,"function":null},"severity":"INFO","time":"2019-03-10T23:13:46.804Z"} �[31m▸ Could not start distribution: {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"talkinto-domain_maint_@10-16-0-4.default.pod.cluster.local", :longnames], false]}, :permanent, 1000, :supervisor, [:erl_distribution]}}
  1. They can race for sys.config file because it looks like ping executes all config providers which overrides config.sys file:
{"could not start kernel pid",application_controller,"error in config file \"/opt/talkinto_domain/var/sys.config\" (none): no ending <dot> found"}
 Unable to configure release! could not start kernel pid (application_controller) (error in config file "/opt/talkinto_domain/var/sys.config" (none): no ending <dot> found)
 
 Crash dump is being written to: erl_crash.dump...done
  1. When used during VM boot (as readiness probe) it would crash the VM because of similar races with VM that runs production code.

@AndrewDryga AndrewDryga changed the title ping command is crashing ping command is crashing because of race conditions Mar 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.