Skip to content
This repository has been archived by the owner on Nov 30, 2021. It is now read-only.

fix(scheduler): container loggers are failing to boot #738

Closed
bacongobbler opened this issue Apr 16, 2014 · 1 comment · Fixed by #766
Closed

fix(scheduler): container loggers are failing to boot #738

bacongobbler opened this issue Apr 16, 2014 · 1 comment · Fixed by #766
Assignees
Labels
Milestone

Comments

@bacongobbler
Copy link
Member

The container loggers are failing on vagrant for some reason. This is probably due to how vagrant handles systemd units locally vs. orchestrating them via fleet. Logs below. cc @gabrtv

$ fleetctl status famous-inventor_v8.web.4-log.service
famous-inventor_v8.web.4-log.service - famous-inventor_v8.web.4 log
   Loaded: loaded (/run/systemd/system/famous-inventor_v8.web.4-log.service; static)
   Active: failed (Result: exit-code) since Wed 2014-04-16 04:10:41 UTC; 3min 33s ago
  Process: 7443 ExecStart=/bin/sh -c /usr/bin/docker logs -f famous-inventor_v8.web.4 2>&1 | logger -p local0.info -t famous-inventor[web.4] --udp --server $(etcdctl get /deis/logs/host | cut -d ':' -f1) --port $(etcdctl get /deis/logs/port | cut -d ':' -f2) (code=exited, status=1/FAILURE)
  Process: 7176 ExecStartPre=/bin/sh -c until /usr/bin/docker inspect famous-inventor_v8.web.4 >/dev/null 2>&1; do sleep 1; done (code=exited, status=0/SUCCESS)
 Main PID: 7443 (code=exited, status=1/FAILURE)

Apr 16 04:10:36 deis systemd[1]: [/run/systemd/system/famous-inventor_v8.web.4-log.service:10] Unknown section 'X-Fleet'. Ignoring.
Apr 16 04:10:36 deis systemd[1]: [/run/systemd/system/famous-inventor_v8.web.4-log.service:11] Assignment outside of section. Ignoring.
Apr 16 04:10:36 deis systemd[1]: [/run/systemd/system/famous-inventor_v8.web.4-log.service:10] Unknown section 'X-Fleet'. Ignoring.
Apr 16 04:10:36 deis systemd[1]: [/run/systemd/system/famous-inventor_v8.web.4-log.service:11] Assignment outside of section. Ignoring.
Apr 16 04:10:36 deis systemd[1]: Starting famous-inventor_v8.web.4 log...
Apr 16 04:10:37 deis systemd[1]: Started famous-inventor_v8.web.4 log.
Apr 16 04:10:37 deis sh[7443]: logger: getaddrinfo Error:100: Name or service not known
Apr 16 04:10:41 deis systemd[1]: famous-inventor_v8.web.4-log.service: main process exited, code=exited, status=1/FAILURE
Apr 16 04:10:41 deis systemd[1]: Unit famous-inventor_v8.web.4-log.service entered failed state.

If it matters, the etcd key /deis/logs does not exist:

$ etcdctl ls /deis
/deis/registry
/deis/cache
/deis/controller
/deis/services
/deis/router
/deis/builder
/deis/database
$ sudo systemctl list-units | grep deis
deis-builder.service                                                                     loaded active running   deis-builder
deis-cache.service                                                                       loaded active running   deis-cache
deis-controller.service                                                                  loaded active running   deis-controller
deis-database.service                                                                    loaded active running   deis-database
deis-logger.service                                                                      loaded active running   deis-logger
deis-registry.service                                                                    loaded active running   deis-registry
deis-router.service                                                                      loaded active running   deis-router
seed-deis-registry.service                                                               loaded failed failed    Seed Deis Registry with Docker Images
@mboersma mboersma added the bug label Apr 16, 2014
@mboersma mboersma added this to the 0.8.0 milestone Apr 16, 2014
@bacongobbler bacongobbler self-assigned this Apr 17, 2014
@bacongobbler
Copy link
Member Author

It appears that the logger is not publishing to etcd. The logger: getaddrinfo Error:100: Name or service not known issue is because it cannot find /deis/logs/host.

$ etcdctl get /deis/logs/host                 
Error: 100: Key not found (/deis/logs) [6815]

bacongobbler pushed a commit that referenced this issue Apr 17, 2014
before, the logger's publish loop would be searching for a TCP
connection. Since the logger runs on UDP, the publish loop would
not find it running and would run forever. This changes netstat
to search over UDP for the logger instead, allowing it to publish
its values to etcd.

fixes #738
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants