Add support for DLO automatic cleanup #278

bastelfreak · 2018-06-01T19:30:11Z

No description provided.

bastelfreak · 2018-06-01T19:30:46Z

This is WIP. Let me know if you're interested in this and I will add tests + README.md updates.

bastelfreak · 2018-06-01T19:33:25Z

Gemfile

  gem 'puppetlabs_spec_helper', :require => false
-  # Pinning due to bug in newer rspec with Ruby 1.8.7
-  gem 'rspec-core', '3.1.7'


I run into some issues while running the tests locally. For debugging I removed the pinning.

We shouldn't need to support such an old Ruby version anymore, so this should be ok.

I rebased against the release branch to get all this fixed. I hope that was okay.

austb

We are still discussing the details around the customization options for cleanup frequency. I'll get back to this PR as soon as I have more information. Thanks for your contributions!

austb · 2018-07-16T19:39:51Z

manifests/server.pp

+      -> systemd::unit_file{'puppetdb-cleanup.timer':
+        content => epp("${module_name}/puppetdb-DLO-cleanup.timer.epp"),
+      }
+      -> service{'puppetdb-cleanup.timer':


This needs to be a ~> so that the service will be restarted if the puppetdb-cleanup.timer file changes.

As an alternative, it looks like the ~> service block can be omitted entirely with something like this (which I pulled from the README)

systemd::unit_file { 'foo.service': source => "puppet:///modules/${module_name}/foo.service", enable => true, active => true, }

After months of work, this feature is finally released. It wasn't available when when I created this PR. updated the code.

austb · 2018-07-16T19:41:05Z

manifests/server.pp

+    if $facts['systemd'] {
+      # deploy a systemd timer + service to cleanup old reports
+      # https://puppet.com/docs/puppetdb/5.2/maintain_and_tune.html#clean-up-the-dead-letter-office
+      systemd::unit_file{'puppetdb-cleanup.service':


We should rename this to puppetdb-dlo-cleanup.service (and the matching timer one below) in case we want to add another cleanup service in the future.

sounds good, done

austb · 2018-07-16T20:14:57Z

manifests/server.pp

+      }
+      -> service{'puppetdb-cleanup.timer':
+        ensure => 'running',
+        enable => true,


I don't think the enable option is necessary here, but I could be mistaken.

It needs to be enabled. Otherwise the timer is off after the next reboot and the upcoming puppet run would start it.

austb · 2018-07-17T21:45:33Z

templates/puppetdb-DLO-cleanup.timer.epp

+
+[Timer]
+# Daily at 8am
+OnCalendar=*-*-* 08:0:00


We should choose the time of day via fqdn_rand. If there are multiple PuppetDB VMs running on the same physical machine this will prevent them from hammering the disk at the same time.

Also the comment will no longer be accurate, so you can remove it or replace it with a general comment about running once a day with a link to documentation on the systemd timer syntax

You could select the hour with fqdn_rand(24) and the minutes with fqdn_rand(60).

austb · 2018-07-17T21:55:32Z

manifests/server.pp

@@ -297,6 +298,23 @@
    }
  }

+  if $automatic_dlo_cleanup {
+    if $facts['systemd'] {


If systemd does not exist, lets try setting up a cron job to clean this up instead. For example Redhat 6 has cron but not systemd.

austb · 2018-07-17T22:03:05Z

@bastelfreak These comments are done and ready for you to take a look at.

austb · 2018-07-17T22:21:48Z

templates/puppetdb-DLO-cleanup.service.epp

+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/find /opt/puppetlabs/server/data/puppetdb/stockpile/discard/ -type f -mtime +90 -delete


Also, one final note. For security reasons, this service should run as the PuppetDB user

added user and group

bastelfreak · 2018-07-19T20:01:58Z

Personal reminder:

should add tests
Update readme
expose maximum DLO age as variable

austb · 2018-07-19T21:33:04Z

Awesome! This looks good. If you need any help writing tests just let us know and we can help out. I would say these would be nice.

Can be disabled (ie. creates neither the service/timer nor the cronjob)
Creates systemd service/timer, and starts timer
Creates cron job if systemd not available
Actually runs as puppetdb user. ie cannot perform some action the PuppetDB user should not be able to do (we can chat about this if it's unclear what to do, I'll have to think about it as well)

And if you add DLO age as variable, a test that that can be customized.

rbrw

Just a few comments with respect to the travis and sh bits. Thanks.

rbrw · 2018-07-23T16:37:15Z

.travis.yml

+  - rm -f Gemfile.lock
+  - gem update --system
+  - gem --version
+  - bundle -v


I'd suggest either moving these to a helper script,

before_install: some/where/prep-tests

or using a different approach, i.e. say with one of the yaml multi-line sytaxes.

For example, using ">":

before_install: > set -ex && bundle -v && rm -f Gemfile.lock && ...

or using "|"

before_install: | set -ex bundle -v rm -f Gemfile.lock ...

(Note that syntax is just approximate -- haven't checked it.)

Otherwise, travis will just keep going iirc. That is, it runs all the bullet points, even if one of them fails, only stopping if there was a failure after it has tried them all.

@rbrw these changes are from the rebase, I'll address them in pr #279

rbrw · 2018-07-23T16:37:19Z

.travis.yml

+  - gem --version
+  - bundle -v
+script:
+  - 'bundle exec rake $CHECK'


Hmm, do we need the single quotes, and regardless, wonder if we should quote the CHECK expansion as "$CHECK", just to avoid having to worry about the content.

@rbrw see above, will address in #279

Zak-Kent · 2018-07-24T17:33:18Z

@bastelfreak, sorry but would you be able to rebase this on master again? We ended up making some changes/rebasing the existing pr we had up before it merged. Thanks for all the help!

bastelfreak · 2018-07-24T18:02:38Z

@austb can you review this again please?

There is also a small change to a string that needed to be double quoted.

austb · 2018-07-24T20:16:55Z

Thanks @bastelfreak for the PR!

bastelfreak commented Jun 1, 2018

View reviewed changes

bastelfreak mentioned this pull request Jul 14, 2018

Setup for 7.0.0 release #279

Merged

austb requested changes Jul 16, 2018

View reviewed changes

austb reviewed Jul 16, 2018

View reviewed changes

austb reviewed Jul 17, 2018

View reviewed changes

bastelfreak force-pushed the systemd branch 2 times, most recently from cd9f333 to 1923209 Compare July 19, 2018 19:55

bastelfreak force-pushed the systemd branch 2 times, most recently from 8b60736 to 7a2bc12 Compare July 20, 2018 20:21

rbrw reviewed Jul 23, 2018

View reviewed changes

bastelfreak force-pushed the systemd branch from 52be38f to 3b8c152 Compare July 24, 2018 17:33

bastelfreak changed the title ~~[WIP] add support for DLO automatic cleanup~~ Add support for DLO automatic cleanup Jul 24, 2018

bastelfreak force-pushed the systemd branch from 3b8c152 to f340561 Compare July 24, 2018 18:00

add support for automatic cleanup

1c25735

bastelfreak force-pushed the systemd branch from f340561 to 1c25735 Compare July 24, 2018 18:02

Add tests for automatic DLO cleanup

df48592

There is also a small change to a string that needed to be double quoted.

austb approved these changes Jul 24, 2018

View reviewed changes

austb merged commit 4e4e81a into puppetlabs:master Jul 24, 2018

bastelfreak deleted the systemd branch July 24, 2018 21:34

smortex added the enhancement label Apr 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for DLO automatic cleanup #278

Add support for DLO automatic cleanup #278

bastelfreak commented Jun 1, 2018

bastelfreak commented Jun 1, 2018

bastelfreak Jun 1, 2018

austb Jul 16, 2018

bastelfreak Jul 19, 2018

austb left a comment

austb Jul 16, 2018 •

edited

Loading

bastelfreak Jul 19, 2018

austb Jul 16, 2018

bastelfreak Jul 19, 2018

austb Jul 16, 2018 •

edited

Loading

bastelfreak Jul 19, 2018

austb Jul 17, 2018

austb Jul 17, 2018

austb Jul 17, 2018

bastelfreak Jul 24, 2018

austb commented Jul 17, 2018

austb Jul 17, 2018

bastelfreak Jul 19, 2018

bastelfreak commented Jul 19, 2018

austb commented Jul 19, 2018 •

edited

Loading

rbrw left a comment

rbrw Jul 23, 2018 •

edited

Loading

Zak-Kent Jul 23, 2018

rbrw Jul 23, 2018

Zak-Kent Jul 23, 2018

Zak-Kent commented Jul 24, 2018

bastelfreak commented Jul 24, 2018

austb commented Jul 24, 2018

Add support for DLO automatic cleanup #278

Add support for DLO automatic cleanup #278

Conversation

bastelfreak commented Jun 1, 2018

bastelfreak commented Jun 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

austb left a comment

Choose a reason for hiding this comment

austb Jul 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

austb Jul 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

austb commented Jul 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bastelfreak commented Jul 19, 2018

austb commented Jul 19, 2018 • edited Loading

rbrw left a comment

Choose a reason for hiding this comment

rbrw Jul 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zak-Kent commented Jul 24, 2018

bastelfreak commented Jul 24, 2018

austb commented Jul 24, 2018

austb Jul 16, 2018 •

edited

Loading

austb Jul 16, 2018 •

edited

Loading

austb commented Jul 19, 2018 •

edited

Loading

rbrw Jul 23, 2018 •

edited

Loading