/home/td-agent is missing on Debian stretch

[okkez]

I want to manage plugins based on Gemfile with td-agent3.

$ getent passwd td-agent
td-agent:x:107:109::/home/td-agent:/bin/false

I've added following Gemfile:

source "https://rubygems.org"

gem "fluentd", "1.0.2"
gem "fluent-plugin-rewrite-tag-filter", "2.0.1"

My td-agent.conf is following:

<source>
  @type dummy
  tag dummy.log
</source>

<match dummy.log>
  @type stdout
</match>

I've added /etc/systemd/system/td-agent.service:

[Unit]
Description=td-agent: Fluentd based data collector for Treasure Data
Documentation=https://docs.treasuredata.com/articles/td-agent
After=network-online.target
Wants=network-online.target

[Service]
User=td-agent
Group=td-agent
LimitNOFILE=65536
Environment=LD_PRELOAD=/opt/td-agent/embedded/lib/libjemalloc.so
Environment=GEM_HOME=/opt/td-agent/embedded/lib/ruby/gems/2.4.0/
Environment=GEM_PATH=/opt/td-agent/embedded/lib/ruby/gems/2.4.0/
Environment=FLUENT_CONF=/etc/td-agent/td-agent.conf
Environment=FLUENT_PLUGIN=/etc/td-agent/plugin
Environment=FLUENT_SOCKET=/var/run/td-agent/td-agent.sock
Environment=TD_AGENT_OPTIONS='--gemfile=/etc/td-agent/Gemfile'
PIDFile=/var/run/td-agent/td-agent.pid
RuntimeDirectory=td-agent
Type=forking
ExecStart=/opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS
ExecStop=/bin/kill -TERM ${MAINPID}
ExecReload=/bin/kill -HUP ${MAINPID}
Restart=always
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Stop and start td-agent, and I got following error messages:

-- Unit td-agent.service has begun starting up.
 2月 27 04:42:00 td-agent3 fluentd[6159]: No entry for terminal type "dumb";
 2月 27 04:42:00 td-agent3 fluentd[6159]: using dumb terminal settings.
 2月 27 04:42:00 td-agent3 fluentd[6159]: `/home/td-agent` is not a directory.
 2月 27 04:42:00 td-agent3 fluentd[6159]: Bundler will use `/tmp/bundler/home/vagrant' as your home directory temporarily.
 2月 27 04:42:00 td-agent3 fluentd[6159]: We trust you have received the usual lecture from the local System
 2月 27 04:42:00 td-agent3 fluentd[6159]: Administrator. It usually boils down to these three things:
 2月 27 04:42:00 td-agent3 fluentd[6159]:     #1) Respect the privacy of others.
 2月 27 04:42:00 td-agent3 fluentd[6159]:     #2) Think before you type.
 2月 27 04:42:00 td-agent3 fluentd[6159]:     #3) With great power comes great responsibility.
 2月 27 04:42:00 td-agent3 fluentd[6159]: sudo: no tty present and no askpass program specified
 2月 27 04:42:00 td-agent3 sudo[6164]: pam_unix(sudo:auth): conversation failed
 2月 27 04:42:00 td-agent3 sudo[6164]: pam_unix(sudo:auth): auth could not identify password for [td-agent]
 2月 27 04:42:00 td-agent3 sudo[6164]: td-agent : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/true
 2月 27 04:42:00 td-agent3 fluentd[6159]: Bundler requires sudo access to install at the moment. Try installing again,
 2月 27 04:42:00 td-agent3 fluentd[6159]: granting Bundler sudo access when prompted, or installing into a different path.
 2月 27 04:42:00 td-agent3 systemd[1]: td-agent.service: Control process exited, code=exited status=30
 2月 27 04:42:00 td-agent3 systemd[1]: Failed to start td-agent: Fluentd based data collector for Treasure Data.

Change $HOME and create it:
(This is need to execute bundle install because bundler creates $HOME/.bundle and store repository index cache.)

$ sudo usermod -d /var/lib/td-agent td-agent
$ sudo mkdir -p /var/lib/td-agent
$ sudo chown td-agent:td-agent /var/lib/td-agent

Add sudoers for td-agent with NOPASSWD.

Failed to start td-agent because build-tools are missing.

$ sudo systemctl start td-agent

Install build-essential.

$ sudo systemctl start td-agent

Finally, I can start td-agent with my Gemfile and td-agent.conf.

Is this intentional to need such complicated sequences to manage fluent-plugins with Gemfile?

---

My proposals to manage gems based on Gemfile.

• Change td-agent's home directory to /var/lib/td-agent and create it in package's post install script
  • Generally system user creates home directory under /var/lib
• Run bundle install without sudo
  • Possibly we can run bundle install as td-agent user
  • Maybe we must dive into bundler source code
• Add more documentation about Gemfile based fluent-plugin management to fluentd-docs

[repeatedly]

Change td-agent's home directory to /var/lib/td-agent and create it in package's post install script

We use adduser/useradd for creating td-agent user so change option can do it. AFAIK, rpm creates home directory under /var/lib. Maybe, this problem happens on only deb?

td-agent doesn't consider Gemfile because --gemfileoption is implemented after td-agent release.
Supporting it for recent versions is good idea.

[okkez]

Maybe, this problem happens on only deb?

I've tested on deb for now.

rpm creates $HOME as /var/lib/td-agent explicitly.
https://github.com/treasure-data/omnibus-td-agent/blob/master/templates/package-scripts/td-agent/rpm/post#L7

deb does not create $HOME, but /etc/passwd describes /home/td-agent is $HOME for td-agent user.
https://github.com/treasure-data/omnibus-td-agent/blob/master/templates/package-scripts/td-agent/deb/postinst#L11

td-agent doesn't consider Gemfile because --gemfileoption is implemented after td-agent release.
Supporting it for recent versions is good idea.

Thanks, I will keep investigating supporting --gemfile option.

[okkez]

Investigate on Debian stretch.

Preconditions:

• Change td-agent's $HOME to /var/lib/td-agent
• Add /etc/systemd/system/td-agent.service in first comment
• td-agent user does not have sudo privilege
• Folloing files exist (td-agent config files)
  • /etc/td-agent/Gemfile
  • /etc/td-agent/plugin/
  • /etc/td-agent/td-agent.conf

Result:
If td-agent config files are not writable by td-agent user, bundler needs sudo privilege.
if td-agent config files are writable by td-agent user, bundler does not need sudo privilege.

Additional information, if /etc/td-agent/vendor/bundle that is writable by td-agent user exist, bundler does not need sudo privilege.

We need change owner and group of files under /etc/td-agent directory to manage fluent-plugins by /etc/td-agent/Gemfile.

[okkez]

#168 is merged, so this issue can be closed.