influxdb fails to start on first run

[spuder]

For some reason, the influxdb service fails to start on a puppet apply. I've reproduced this 3 times.

puppet module install jdowning-influxdb
puppet apply /etc/puppet/modules/influxdb/tests/init.pp --debug

The puppet logs show that the service should have been started, yet when you run service influxdb status influx is not running.

Notice: /Stage[main]/Staging/File[/opt/staging]/ensure: created
Debug: /Stage[main]/Staging/File[/opt/staging]: The container Class[Staging] will propagate my refresh event
Debug: Class[Staging]: The container Stage[main] will propagate my refresh event
Notice: /Stage[main]/Influxdb::Install/Staging::File[influxdb-package]/File[/opt/staging/influxdb]/ensure: created
Debug: /Stage[main]/Influxdb::Install/Staging::File[influxdb-package]/File[/opt/staging/influxdb]: The container Staging::File[influxdb-package] will propagate my refresh event
Debug: Exec[/opt/staging/influxdb/influxdb-package](provider=posix): Executing 'curl  -f -L -o /opt/staging/influxdb/influxdb-package http://s3.amazonaws.com/influxdb/influxdb-latest-1.x86_64.rpm'
Debug: Executing 'curl  -f -L -o /opt/staging/influxdb/influxdb-package http://s3.amazonaws.com/influxdb/influxdb-latest-1.x86_64.rpm'
Notice: /Stage[main]/Influxdb::Install/Staging::File[influxdb-package]/Exec[/opt/staging/influxdb/influxdb-package]/returns: executed successfully
Debug: /Stage[main]/Influxdb::Install/Staging::File[influxdb-package]/Exec[/opt/staging/influxdb/influxdb-package]: The container Staging::File[influxdb-package] will propagate my refresh event
Debug: Staging::File[influxdb-package]: The container Class[Influxdb::Install] will propagate my refresh event
Debug: Prefetching rpm resources for package
Debug: Executing '/bin/rpm --version'
Debug: Executing '/bin/rpm -qa --nosignature --nodigest --qf '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}\n''
Debug: Executing '/bin/rpm -q influxdb --nosignature --nodigest --qf %{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}\n'
Debug: Executing '/bin/rpm -i /opt/staging/influxdb/influxdb-package'
Notice: /Stage[main]/Influxdb::Install/Package[influxdb]/ensure: created
Debug: /Stage[main]/Influxdb::Install/Package[influxdb]: The container Class[Influxdb::Install] will propagate my refresh event
Notice: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_lru_cache_size]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_lru_cache_size]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_lru_cache_size]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_max_backoff]/value: value changed '"10s" # the maxmimum backoff after a failed heartbeat attempt' to '"10s"'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_max_backoff]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_max_backoff]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_timeout]/value: value changed '"2s" # the write timeout on the protobuf conn any duration parseable by time.ParseDuration' to '"2s"'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_timeout]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_timeout]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[wal_bookmark_after]/value: value changed '1000 # the number of writes after which a bookmark will be created' to '0'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[wal_bookmark_after]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[wal_bookmark_after]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[cluster_seed_servers]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[cluster_seed_servers]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[cluster_seed_servers]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[logging_file]/value: value changed '"/opt/influxdb/shared/log.txt"         # stdout to log to standard out' to '"/opt/influxdb/shared/influxdb.log"'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[logging_file]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[logging_file]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[api_port]/value: value changed '8086    # binding is disabled if the port isn't set' to '8086'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[api_port]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[api_port]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[admin_port]/value: value changed '8083              # binding is disabled if the port isn't set' to '8083'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[admin_port]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[admin_port]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_heartbeat]/value: value changed '"200ms" # the heartbeat interval between the servers. must be parseable by time.ParseDuration' to '"200ms"'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_heartbeat]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_heartbeat]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[raft_election_timeout]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[raft_election_timeout]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[raft_election_timeout]: Scheduling refresh of Service[influxdb]
Debug: Class[Influxdb::Install]: The container Stage[main] will propagate my refresh event
Notice: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_point_write_size]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_point_write_size]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_point_write_size]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[cluster_write_buffer_size]/value: value changed '1000' to '10000'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[cluster_write_buffer_size]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[cluster_write_buffer_size]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_min_backoff]/value: value changed '"1s" # the minimum backoff after a failed heartbeat attempt' to '"1s"'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_min_backoff]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[cluster_protobuf_min_backoff]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[hostname]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[hostname]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[hostname]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_max_open_files]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_max_open_files]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_max_open_files]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_point_batch_size]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_point_batch_size]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_point_batch_size]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[wal_flush_after]/value: value changed '1000 # the number of writes after which wal will be flushed, 0 for flushing on every write' to '0'
Debug: /Stage[main]/Influxdb::Config/Ini_setting[wal_flush_after]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[wal_flush_after]: Scheduling refresh of Service[influxdb]
Notice: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_max_open_shards]/ensure: created
Debug: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_max_open_shards]: The container Class[Influxdb::Config] will propagate my refresh event
Info: /Stage[main]/Influxdb::Config/Ini_setting[leveldb_max_open_shards]: Scheduling refresh of Service[influxdb]
Debug: Class[Influxdb::Config]: The container Stage[main] will propagate my refresh event
Debug: Executing '/sbin/service influxdb status'
Debug: Executing '/sbin/chkconfig influxdb'
Debug: Executing '/sbin/service influxdb start'
Debug: Executing '/sbin/chkconfig influxdb'
Notice: /Stage[main]/Influxdb::Service/Service[influxdb]/ensure: ensure changed 'stopped' to 'running'
Debug: /Stage[main]/Influxdb::Service/Service[influxdb]: The container Class[Influxdb::Service] will propagate my refresh event
Info: /Stage[main]/Influxdb::Service/Service[influxdb]: Unscheduling refresh on Service[influxdb]
Debug: Class[Influxdb::Service]: The container Stage[main] will propagate my refresh event
Debug: Finishing transaction 70299674771260
Debug: Storing state
Debug: Stored state in 0.07 seconds
Notice: Finished catalog run in 13.39 seconds
Debug: Using settings: adding file resource 'rrddir': 'File[/var/lib/puppet/rrd]{:ensure=>:directory, :links=>:follow, :group=>"puppet", :backup=>false, :owner=>"puppet", :mode=>"750", :path=>"/var/lib/puppet/rrd", :loglevel=>:debug}'
Debug: Finishing transaction 70299673947040
Debug: Received report to process from influxdb-test.vapor2.ac
Debug: Processing report from influxdb-test.vapor2.ac with processor Puppet::Reports::Store
[root@influxdb-test ~]# service influxdb status
influxdb Process is not running [ FAILED ]
[root@influxdb-test ~]# ps aux |grep influxdb
root      1677  0.0  0.0 103244   868 pts/1    S+   16:53   0:00 grep influxdb
[root@influxdb-test ~]# service influxdb status
influxdb Process is not running [ FAILED ]

I suspect this is either an ording issue (where the service isn't the very last thing in the catalog to be run), or the service needs a special 'hasstatus' line.

[spuder]

Additional Datapoint:

If you manually stop the influxdb service, then manually run puppet, it also fails to start.

service influxdb stop
puppet apply /etc/puppet/modules/influxdb/tests/init.pp --debug

Debug: Prefetching rpm resources for package
Debug: Executing '/bin/rpm --version'
Debug: Executing '/bin/rpm -qa --nosignature --nodigest --qf '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH}\n''
Debug: Executing '/sbin/service influxdb status'
Debug: Executing '/sbin/chkconfig influxdb'
Debug: Executing '/sbin/service influxdb start'
Debug: Executing '/sbin/chkconfig influxdb'
Notice: /Stage[main]/Influxdb::Service/Service[influxdb]/ensure: ensure changed 'stopped' to 'running'
Debug: /Stage[main]/Influxdb::Service/Service[influxdb]: The container Class[Influxdb::Service] will propagate my refresh event
Info: /Stage[main]/Influxdb::Service/Service[influxdb]: Unscheduling refresh on Service[influxdb]
Debug: Class[Influxdb::Service]: The container Stage[main] will propagate my refresh event
Debug: Finishing transaction 70335002674860
Debug: Storing state
Debug: Stored state in 0.15 seconds
Notice: Finished catalog run in 0.72 seconds
Debug: Using settings: adding file resource 'rrddir': 'File[/var/lib/puppet/rrd]{:ensure=>:directory, :links=>:follow, :group=>"puppet", :backup=>false, :owner=>"puppet", :mode=>"750", :path=>"/var/lib/puppet/rrd", :loglevel=>:debug}'
Debug: Finishing transaction 70335002375260
Debug: Received report to process from influxdb-test.vapor2.ac
Debug: Processing report from influxdb-test.vapor2.ac with processor Puppet::Reports::Store
[root@influxdb-test ~]# ps aux |grep influxdb
root      1861  0.0  0.0 103244   864 pts/1    S+   17:00   0:00 grep influxdb
[root@influxdb-test ~]# service influxdb status
influxdb Process is not running [ FAILED ]

[spuder]

I tried fixing the init script to conform to LSB standards, with no luck.
Strangely influxdb does start when inside the vagrant vm.

Turns out this is a problem with the init script in influxdb. It uses sudo which won't work if your /etc/sudoers file contains requretty.
influxdata/influxdb#800

The vagrant box provided by puppetlabs has requiretty commented out. Explains why it worked in Vagrant, but not in any other VM that I used.