New module with Github actions #8

jsirianni · 2021-06-23T20:04:42Z

What does this PR accomplish?

This PR is at the request of @qingling128. This module brings Puppet support for the Google Ops Agent and more (see README.md for details).

Continuous Integration

Github actions will test all agents on Linux and the Ops Agent on Windows.

  Service google-cloud-ops-agent.target
     ✔  is expected to be installed
     ✔  is expected to be enabled
     ✔  is expected to be running
  System Package google-cloud-ops-agent
     ✔  is expected to be installed
     ✔  version is expected to match /1.0.4/
  File /etc/google-cloud-ops-agent/config.yaml
     ✔  is expected to exist

A handful of prerequisites are required before CI will finish successfully. I can assist in this area. See test/README.md for specific details.

* replace repo with new puppet module * update workflows to reflect new branch names

* Add windows ci * fix branch * add windows image * rename because we now support windows * test windows

* switch from 1.0.4 to 1.0.10 on windows, the agent stop bug is resolved sometime after 1.0.4 * test 1.0.10 in github actions * now that stopping a service works, re-add custom_config testing * fix path to source config * Fix resource name typo * try running as system user. this command should work but fails in ci. it works under rdp when running 'as administrator' * revert back to command prompt now that the new agent can stop currectly. powershell syntax keeps failing for some reason * revert user: SYSTEM. puppet on windows cannot switch users, but I have comfirmed that the ci user should be able to stop services without being elevated * sleep before restarting the agent to avoid a race condition where we restart during the intiail agent startup (results in a windows error) * puppet does not like the >NUL param * fix inspec test now that puppet windows restarts are working * checksum seems to change on the windows filesystem, test the checksum for install and custom_install, using the hash returned from inspec failures * correct checksum for install

qingling128

Overall LGTM. Left some minor documentation / testing related comments. Most of them are not blocking this PR (I tried to make it clear by calling them out).

.github/workflows/linux.yml

.github/scripts/inspec.sh

manifests/agent.pp

metadata.json

qingling128 · 2021-06-25T01:26:05Z

test/README.md

+
+Packer will require access to Google APIs. This [guide provides](https://cloud.google.com/build/docs/building/build-vm-images-with-packer) examples on how to setup your working environment.
+
+### Linux


Are these images built once and hosted somewhere, or built from scratch every time during a test run? I guess for Windows it is probably hosted somewhere, since it's built manually, right?

I pushed a commit clarifying this. Basically, Packer is used to prepare base images for Linux. The same process is done manually for Windows. Terraform will deploy these "baked" images for Linux and Windows tests every time CI runs. The Windows instances are not left behind to be re-used, they are created on demand.

The project being used for CI should own these images, or have read access to them if they are in a different project.

qingling128 · 2021-06-25T01:31:50Z

test/cases/linux/ops-agent/1.0.4/custom_config/config.yaml

@@ -0,0 +1,45 @@
+# Copyright 2020 Google LLC


Are the test files for 1.0.4 maintained by some automation system (aka auto generated), or entered by hand?

They are not generated, created by hand. This is okay for now but more complex CI should probably have some sort of generation process.

test/cases/linux/ops-agent/1.0.4/uninstall/manifests/site.pp

qingling128 · 2021-06-25T01:42:42Z

test/cases/linux/logging/latest/custom_config/files/plugins/plugin.rb

@@ -0,0 +1,17 @@
+module Fluent


If we are looking for a sample fluentd config for 3rd party app (In some cases we call it plugin but it's actually not very accurate), here is an example: https://github.com/GoogleCloudPlatform/fluentd-catch-all-config/blob/master/configs/config.d/apache.conf

Understood, I have implemented the config used here https://github.com/GoogleCloudPlatform/google-cloud-ops-agents-ansible/blob/master/molecule/resources/example_configs/plugins/custom_config.conf

* check source header for apache 2.0 license * use script to check for invalid headers * on pr and commit to master * resolve shellcheck concerns * fix name * install go and install addlicense with go install * fix syntax * add Google license headers * Fix linux config checksums, they are different because of the license addition

* Add context to inspec and puppet scripts. Remove project and zone variables from inspec / puppet scripts as they are no longer used. * Add context to setup.sh * Add context to ssh.sh * Add context to terraform wrapper scripts * project is still required for the action

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

* rename cloud_ops to google_cloud_ops * update module path to reflect new module name * use latest checksum for config

* try 2.0.0 in ci * update test cases to support GA 2.0.0 release * Remove support for pre ga * update to ga service name * Remove support for pre ga * fix windows version typo * test against new service names * use new checksum for default config * use 2.0.0 config * use correct checksums * switch to 2.0.1, resolve issue where systemd service is 'not running' * use 2.0.1 for example version * use 2.0.1 for example version * fix config path * fix config path * fix checksum due to new path * fix 2.0.1 service name * temp disable some runs to save time * detect service name based on ops agent version * try 1.0.10 with 2.0.1 upgrade * Add 1.0.10 * try split builtin * remove 1.x.x support. test 2.0.1 on windows * test windows 2.0.1 * enable all

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

* pr feedback * run ci on changes to metadata.json

* use custom config, not plugin * test logging * fix checksum for new test config * re-enable all

qingling128

LGTM

…g Agent and Monitoring Agent, and integrate with Github actions. (#8)

qingling128 · 2021-07-14T23:36:16Z

Just merged this PR.

Looks like some integration tests failed. Might need some investigation.

BTW, I just noticed the last commit of the PR did not trigger integration tests:

I guess some of the commits that addressed code review feedback introduced a breakage somewhere, while the file is not captured by: https://github.com/GoogleCloudPlatform/puppet-google-logging/blob/ff1be4759a86a4a46c8f2c5b7bf961fce017f0ec/.github/workflows/linux.yml#L5

On the side, we are also looking for Puppet experts inside Google to see if there are any Puppet conventions we need to follow within Google or in general. If there is any feedback there, we can send follow up PRs to address them.

jsirianni · 2021-07-14T23:38:45Z

That is expected. The repo does not have secrets created. We should talk about next steps for implementing CI against a GCP project.

…

On Wed, Jul 14, 2021, 19:36 Ling Huang ***@***.***> wrote: Just merged this PR. Looks like some integration tests failed. Might need some investigation. [image: image] <https://user-images.githubusercontent.com/5287526/125705847-8d458f63-1f72-435f-95fd-32f4e6bb7fe1.png> BTW, I just noticed the last commit of the PR did not trigger integration tests: [image: image] <https://user-images.githubusercontent.com/5287526/125705858-41d1d7f3-c49b-4ab8-adc0-38064cdcf311.png> I guess some of the commits that addressed code review feedback introduced a breakage somewhere, while the file is not captured by: https://github.com/GoogleCloudPlatform/puppet-google-logging/blob/ff1be4759a86a4a46c8f2c5b7bf961fce017f0ec/.github/workflows/linux.yml#L5 On the side, we are also looking for Puppet experts inside Google to see if there are any Puppet conventions we need to follow within Google or in general. If there is any feedback there, we can send follow up PRs to address them. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFPZ57AJOJURZCEYP4DWTW3TXYNPXANCNFSM47GPSMRQ> .

jsirianni added 4 commits June 22, 2021 11:07

Replace (#1)

55c7b96

* replace repo with new puppet module * update workflows to reflect new branch names

fix badge links

1b94746

Add windows ci (#2)

a1b2a21

* Add windows ci * fix branch * add windows image * rename because we now support windows * test windows

qingling128 reviewed Jun 25, 2021

View reviewed changes

jsirianni and others added 12 commits June 30, 2021 14:24

re-enable 1.0.4 ops agent tests on linux (#5)

926bb10

Update README.md

6c4ce7a

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

Update README.md

c36e77a

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

Update README.md

544c5f5

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

Update README.md

b36acb9

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

Update README.md

7793d12

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

Update README.md

f845feb

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

Update README.md

218620e

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

use package_state string instead of installed bool (#7)

1b833d9

rename cloud_ops to google_cloud_ops (#8)

3b15751

* rename cloud_ops to google_cloud_ops * update module path to reflect new module name * use latest checksum for config

This was referenced Jul 1, 2021

Add tests for MAJOR.*.* version use cases. #9

Open

Add tests for more Windows distros #10

Open

Add documentation for release instruction and local development workflow #11

Open

jsirianni and others added 5 commits July 7, 2021 15:38

Update README.md

4980ac2

Co-authored-by: Ling Huang <qingling128@users.noreply.github.com>

pr feedback (#10)

16178da

* pr feedback * run ci on changes to metadata.json

clarify packer images

2537f31

Logging additional config (#11)

d809256

* use custom config, not plugin * test logging * fix checksum for new test config * re-enable all

qingling128 approved these changes Jul 13, 2021

View reviewed changes

move create image step to end

bea4600

qingling128 merged commit ff1be47 into GoogleCloudPlatform:master Jul 14, 2021

qingling128 pushed a commit that referenced this pull request Jul 14, 2021

Replace the old module with a new module to support Ops Agent, Loggin…

9391246

…g Agent and Monitoring Agent, and integrate with Github actions. (#8)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New module with Github actions #8

New module with Github actions #8

jsirianni commented Jun 23, 2021

qingling128 left a comment

qingling128 Jun 25, 2021

jsirianni Jul 7, 2021

qingling128 Jun 25, 2021

jsirianni Jul 7, 2021

qingling128 Jun 25, 2021

jsirianni Jul 7, 2021

qingling128 left a comment

qingling128 commented Jul 14, 2021

jsirianni commented Jul 14, 2021 via email


		Packer will require access to Google APIs. This [guide provides](https://cloud.google.com/build/docs/building/build-vm-images-with-packer) examples on how to setup your working environment.

		### Linux

New module with Github actions #8

New module with Github actions #8

Conversation

jsirianni commented Jun 23, 2021

What does this PR accomplish?

Continuous Integration

qingling128 left a comment

Choose a reason for hiding this comment

qingling128 Jun 25, 2021

Choose a reason for hiding this comment

jsirianni Jul 7, 2021

Choose a reason for hiding this comment

qingling128 Jun 25, 2021

Choose a reason for hiding this comment

jsirianni Jul 7, 2021

Choose a reason for hiding this comment

qingling128 Jun 25, 2021

Choose a reason for hiding this comment

jsirianni Jul 7, 2021

Choose a reason for hiding this comment

qingling128 left a comment

Choose a reason for hiding this comment

qingling128 commented Jul 14, 2021

jsirianni commented Jul 14, 2021 via email