sample-data-service mock infra client data #4398

kvivek1115 · 2020-10-12T13:08:35Z

🔩 Description: What code changed, and why?

This PR only responsible to run sample-data-service using various deploy options.
Define sample-data-service as product and user can run this service using --product sample-data.

⛓️ Related Resources

#4141

👍 Definition of Done

sample-data-service is enabled and running.

👟 How to Build and Test the Change

rebuild components/sample-data-service
rebuild components/automate-deployment
in hab studio:

enable_sample_data_service

OR

> cat sample-data.toml 
[deployment.v1.svc]
      products = ["sample-data", "automate"]

> chef-automate config patch sample-data.toml

OR

> chef-automate deploy --product sample-data

check sample-data-service status chef-automate status

sample-data-service     running        ok            12204      4977

✅ Checklist

I have read the CONTRIBUTING document.
Tests added/updated?
Docs added/updated?
All commits have been signed-off for the Developer Certification of Origin.

📷 Screenshots, if applicable

netlify · 2020-10-12T13:08:40Z

Deploy preview for chef-automate ready!

Built with commit 7565e9e

https://deploy-preview-4398--chef-automate.netlify.app

stevendanna · 2020-10-14T08:53:32Z

products.meta

+      "name": "sample-data",
+      "type": "product",
+      "dependencies": ["automate"],
+      "services": []


I think we'll want to list the new services in this service array as chef/sample-data-service

kvivek1115 · 2020-10-23T08:54:48Z

build fc35c99adf07 is failing due to:


./testssl.sh --parallel --quiet -n none --color 0 --severity LOW --warnings off --disable-rating --jsonfile=/tmp/testssl.lrn6Gn/jsonfile_00000037.json --fast --add-ca /hab/svc/deployment-service/data/root.crt --warnings=batch 127.0.0.1:10155
--
  |  
  | Start 2020-10-22 11:17:02        -->> 127.0.0.1:10155 (127.0.0.1) <<--
  |  
  | rDNS (127.0.0.1):       (instructed to minimize DNS queries)./testssl.sh: connect: Connection refused
  | ./testssl.sh: line 10817: /dev/tcp/127.0.0.1/10155: Connection refused
  | Oops: TCP connect problem
  |  
  | Unable to open a socket to 127.0.0.1:10155.
  | Fatal error: Can't connect to "127.0.0.1:10155"
  | Make sure a firewall is not between you and your scanning target!
  |  
  |  
  | /go/src/github.com/chef/automate
  | /usr/bin/buildkite-agent
  | 2020-10-22 11:17:41 INFO   Found 1 files that match "/tmp/tmp.Z70e6hNAQk"
  | 2020-10-22 11:17:41 INFO   Creating (0-1)/1 artifacts
  | 2020-10-22 11:17:41 INFO   Uploading artifact ec9a32da-7c90-48ba-8db8-f913e33e796b tmp/tmp.Z70e6hNAQk (54525 bytes)
  | [Thu Oct 22 11:17:43 UTC 2020][error] == SSL CONFIGURATION TEST FAILED ==
  | [Thu Oct 22 11:17:43 UTC 2020][error] Found 1 SSL config violations not whitelisted by ssl_filters.jq file
  | [Thu Oct 22 11:17:43 UTC 2020][error] Test failures may be caused by changes to https://github.com/drwetter/testssl.sh
  | [Thu Oct 22 11:17:43 UTC 2020][error] testssl.sh was last changed on Tue Oct 20 21:00:02 2020 +0200
  | [Thu Oct 22 11:17:43 UTC 2020][error] Violations (excluding whitelisted):
  | [
  | {
  | "id": "scanProblem",
  | "ip": "127.0.0.1/127.0.0.1",
  | "port": "10155",
  | "severity": "FATAL",
  | "finding": "Can't connect to '127.0.0.1:10155'\nMake sure a firewall is not between you and your scanning target!"
  | }
  | ]
  | /usr/bin/buildkite-agent
  | Deploy Failed - Uploading logs to buildkite
  | 2020-10-22 11:17:44 INFO   Found 3 files that match "logs/*"
  | 2020-10-22 11:17:44 INFO   Creating (0-3)/3 artifacts
  | 2020-10-22 11:17:44 INFO   Uploading artifact 6642c7f7-8f32-44f4-a1cf-f9d2c16862a9 logs/all (1874609 bytes)
  | 2020-10-22 11:17:44 INFO   Uploading artifact f031baf2-b17b-4740-be2f-d42fae9aa514 logs/deploy (896108 bytes)
  | 2020-10-22 11:17:44 INFO   Uploading artifact 79448e20-02b4-469d-9545-33e50b9c19b3 logs/remaining (978931 bytes)
  | [Thu Oct 22 11:17:46 UTC 2020][info] Killing background processes
  | [Thu Oct 22 11:17:46 UTC 2020][info] Exiting with 1
  | Attempting to clean up docker container automate-24792-4256
  | automate-24792-4256
  | ------------------------------------------------------------
  | To run this test locally:
  |  
  | integration/run_test integration/tests/security.sh
  |  
  | ------------------------------------------------------------
  | 🚨 Error: The command exited with status 1
  |

srenatus · 2020-10-23T09:28:59Z

It looks like the testssl call, which checks all open ports, cannot connect to the new service, sample-data-service.

Is it supposed to be running at all times? Or something you selectively start or do not start? I guess that either the port should be excluded from the scan targets or we should check that it really comes online. 🤔

srenatus · 2020-10-23T09:29:36Z

Nevermind, the description says that you have to select it. I guess the testssl deployment (i.e. the "security" test run) doesn't do it.

srenatus · 2020-10-23T09:31:59Z

@vsingh-msys I've pushed a commit enabling the sample-data product for the security test. Let's see what happens now 😄

srenatus · 2020-10-23T09:53:48Z

Looks like the security test now is green. ✔️

kvivek1115 · 2020-10-23T10:00:30Z

Yeah, thanks @srenatus

kvivek1115 · 2020-10-29T11:29:47Z

components/sample-data-service/cmd/sample-data-service/commands/serve.go

+	if err != nil {
+		fail(errors.Wrap(err, "Could not read certs"))
+	}
+	connFactory := secureconn.NewFactory(*serviceCerts)


Added service to get started maybe we can skip it later

lancewf

Working for me. The code looks good also.
Thank you for breaking this task up to only adding the service first.

stevendanna

This seems like a reasonable start to me! Nice work. Left mostly minor comments.

stevendanna · 2020-11-03T09:17:18Z

api/config/sample_data/config_request.proto

+			Log log = 4;
+
+			message Service {
+				google.protobuf.StringValue host = 1 [deprecated=true]; // The listen host is no longer setable(localhost only)


We can avoid adding a deprecated option at the start of this service since no users have used this option yet.

stevendanna · 2020-11-03T09:18:02Z

components/sample-data-service/README.md

@@ -0,0 +1 @@
+# Sample data service


Could we add a bit of content here? What does the service do? How do you use it?

stevendanna · 2020-11-03T09:20:21Z

components/sample-data-service/server/server.go

+		Sha:     version.GitSHA,
+		Built:   version.BuildTime,
+	}, nil
+}


You can get this by default via the debug API.

Noted thanks!

- Add required config options Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

We still need configuration protobufs for this service. Signed-off-by: Steven Danna <steve@chef.io>

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

msorens · 2020-11-03T15:24:42Z

.studio/sample-data-service

+# Helper methods specific for the sample-data-service
+
+document "start_sample_data_service" <<DOC
+  Build and start the local sample-data-service


Let's be consistent. Here you use "sample-data-service" and on the next function you use "Sample data service", and "sample data service" in other files. Suggest stick with "sample data service".

thanks for nitpick suggestion!

msorens · 2020-11-03T15:25:33Z

.studio/sample-data-service

+  install_if_missing core/jq-static jq
+  install_if_missing core/grpcurl grpcurl


Are these necessary here? You are not using either one.

Yes, starting the service alone and installing grpcurl & jq would be handy so that grpcurl & jq can be used in hab studio.

msorens · 2020-11-03T15:27:19Z

api/config/sample_data/config_request.go

+func DefaultConfigRequest() *ConfigRequest {
+	c := NewConfigRequest()
+	c.V1.Sys.Service.Host = w.String("127.0.0.1")
+	c.V1.Sys.Service.Port = w.Int32(10155)


How did you determine this value?

We can pick any of the port which is available in chef-automate dev ports free

Cool; did not know that command!

msorens · 2020-11-03T15:48:30Z

components/sample-data-service/Makefile

+  verbose?="-v"
+endif
+
+all: lint build test


The test target is missing? Are there tests to run? If not, remove it here.

Right now, have added these helpers to the future building, removing test make sense we can add these helpers once required. Thanks!

msorens · 2020-11-03T15:49:18Z

components/sample-data-service/Makefile

+	mkdir -p bin/
+	go build -o bin/sample-data-service ./cmd/sample-data-service
+
+.PHONY: all static unit build compile proto test dep-ensure have-dep lint


Suggested change

.PHONY: all static unit build compile proto test dep-ensure have-dep lint

.PHONY: all static unit build test lint

(And possibly remove test as well per last comment.)

msorens · 2020-11-03T15:50:40Z

components/sample-data-service/README.md

@@ -0,0 +1,5 @@
+# Sample data service
+
+Sample data service can be used to send scheduled data to Automate mimicing the data that would be send from a client.


Suggested change

Sample data service can be used to send scheduled data to Automate mimicing the data that would be send from a client.

Sample data service can be used to send scheduled data to Automate mimicking the data that would be send from a client.

msorens · 2020-11-03T15:52:49Z

components/sample-data-service/server/server.go

+	"github.com/chef/automate/lib/tracing"
+)
+
+// Server is an sample-data server


Suggested change

// Server is an sample-data server

// Server is a sample-data server

msorens · 2020-11-03T15:53:04Z

components/sample-data-service/server/server.go

+	service *service.Service
+}
+
+// NewServer returns an sample-data server


Suggested change

// NewServer returns an sample-data server

// NewServer returns a sample-data server

msorens · 2020-11-03T15:54:47Z

components/sample-data-service/server/server.go

+	return &Server{service: service}
+}
+
+// NewGRPCServer creates a grpc server that serves all Teams GRPC APIs


copy-pasted stuff forgot to update comment thanks!

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

kvivek1115 · 2020-11-09T05:54:45Z

Hoping everything is as expected, merging this PR so proceeding with the further roadmap. 🤞

author Michael Sorens <msorens@chef.io> 1604676405 -0800 committer Alex Pop <apop@chef.io> 1605705520 +0000 Pin semgrep for custom rules (#4426) sample-data-service mock infra client data (#4398) Co-authored-by: Steven Danna <steve@chef.io> Compliance APIs default to 24h without end_time (#4310) * Compliance APIs default to 24h without end_time Signed-off-by: Alex Pop <apop@chef.io> Fix overlapping Signed-off-by: Alex Pop <apop@chef.io> * Update docker test based on new inspec error output Signed-off-by: Alex Pop <apop@chef.io> * Remove fmt package Signed-off-by: Alex Pop <apop@chef.io> Add option to select last 24h compliance report (#4358) * Add option to select last 24h compliance report This commit adds the option to view the last 24h of compliance data. Compliance API endpoints were recently updated to return last 24h window of data if `start_time` and `end_time` are excluded from requests. Signed-off-by: Scott Christopherson <scott@chef.io> * Align filters Signed-off-by: Alex Pop <apop@chef.io> * Update docker test based on new inspec error output Signed-off-by: Alex Pop <apop@chef.io> Co-authored-by: Alex Pop <apop@chef.io> Add Compliance Knowledge Transfer notes Signed-off-by: Alex Pop <apop@chef.io> small updates Signed-off-by: Alex Pop <apop@chef.io> before call Signed-off-by: Alex Pop <apop@chef.io> before call Signed-off-by: Alex Pop <apop@chef.io> added filters and migrations added some filter types ingest clarifications Signed-off-by: Alex Pop <apop@chef.io> Before second kt call Signed-off-by: Alex Pop <apop@chef.io> changes post second call Signed-off-by: Alex Pop <apop@chef.io>

author Michael Sorens <msorens@chef.io> 1604676405 -0800 committer Alex Pop <apop@chef.io> 1605705520 +0000 Pin semgrep for custom rules (#4426) sample-data-service mock infra client data (#4398) Co-authored-by: Steven Danna <steve@chef.io> Compliance APIs default to 24h without end_time (#4310) * Compliance APIs default to 24h without end_time Signed-off-by: Alex Pop <apop@chef.io> Fix overlapping Signed-off-by: Alex Pop <apop@chef.io> * Update docker test based on new inspec error output Signed-off-by: Alex Pop <apop@chef.io> * Remove fmt package Signed-off-by: Alex Pop <apop@chef.io> Add option to select last 24h compliance report (#4358) * Add option to select last 24h compliance report This commit adds the option to view the last 24h of compliance data. Compliance API endpoints were recently updated to return last 24h window of data if `start_time` and `end_time` are excluded from requests. Signed-off-by: Scott Christopherson <scott@chef.io> * Align filters Signed-off-by: Alex Pop <apop@chef.io> * Update docker test based on new inspec error output Signed-off-by: Alex Pop <apop@chef.io> Co-authored-by: Alex Pop <apop@chef.io> Add Compliance Knowledge Transfer notes Signed-off-by: Alex Pop <apop@chef.io> small updates Signed-off-by: Alex Pop <apop@chef.io> before call Signed-off-by: Alex Pop <apop@chef.io> before call Signed-off-by: Alex Pop <apop@chef.io> added filters and migrations added some filter types ingest clarifications Signed-off-by: Alex Pop <apop@chef.io> Before second kt call Signed-off-by: Alex Pop <apop@chef.io> changes post second call Signed-off-by: Alex Pop <apop@chef.io> Co-authored-by: Michael Sorens <msorens@chef.io>

kvivek1115 self-assigned this Oct 12, 2020

kvivek1115 changed the title ~~sample-data-service mock infra client data~~ [WIP] sample-data-service mock infra client data Oct 12, 2020

stevendanna reviewed Oct 14, 2020

View reviewed changes

kvivek1115 force-pushed the VSingh/sample-data-service branch from 471278f to 305d2da Compare October 22, 2020 10:57

kvivek1115 changed the title ~~[WIP] sample-data-service mock infra client data~~ sample-data-service mock infra client data Oct 22, 2020

kvivek1115 added the sample-data-service Sample data service mock chef infra client runs label Oct 22, 2020

kvivek1115 marked this pull request as ready for review October 22, 2020 14:02

kvivek1115 requested review from btm, stevendanna, lancewf and srenatus October 27, 2020 15:26

kvivek1115 commented Oct 29, 2020

View reviewed changes

lancewf approved these changes Oct 29, 2020

View reviewed changes

stevendanna reviewed Nov 3, 2020

View reviewed changes

Vivek Singh and others added 9 commits November 3, 2020 15:39

- Add new service sample-data-service

362c3fd

- Add required config options Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

Wire sample-data-service into deployment

40598f3

We still need configuration protobufs for this service. Signed-off-by: Steven Danna <steve@chef.io>

config request API gen

6f1628d

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

deployment automate config

5673797

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

make generate & bindings

557d00a

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

Add gRPC server & service

0b0c882

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

minor changes

e17af85

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

verify build & update bindings

b62c060

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

missing new line

d1c546d

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

More detailed README.md, removed deprecated config host comment.

b94ab3c

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

kvivek1115 force-pushed the VSingh/sample-data-service branch from f56c620 to b94ab3c Compare November 3, 2020 10:10

msorens reviewed Nov 3, 2020

View reviewed changes

Vivek Singh added 4 commits November 4, 2020 06:33

typo & minor reviews comments

ebb016a

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

.bldr.toml changes

74d0e8e

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

unit test case from .expeditor/verify.pipeline.yml

efc506a

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

testssl security.sh upadates for sample-data service

7565e9e

Signed-off-by: Vivek Singh <vivek.singh@msystechnologies.com>

kvivek1115 merged commit 3d88cc0 into master Nov 9, 2020

kvivek1115 deleted the VSingh/sample-data-service branch November 9, 2020 05:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sample-data-service mock infra client data #4398

sample-data-service mock infra client data #4398

kvivek1115 commented Oct 12, 2020 •

edited

Loading

netlify bot commented Oct 12, 2020 •

edited

Loading

stevendanna Oct 14, 2020

kvivek1115 commented Oct 23, 2020

srenatus commented Oct 23, 2020

srenatus commented Oct 23, 2020

srenatus commented Oct 23, 2020

srenatus commented Oct 23, 2020

kvivek1115 commented Oct 23, 2020

kvivek1115 Oct 29, 2020

lancewf left a comment

stevendanna left a comment

stevendanna Nov 3, 2020

stevendanna Nov 3, 2020

stevendanna Nov 3, 2020

kvivek1115 Nov 3, 2020

msorens Nov 3, 2020 •

edited

Loading

kvivek1115 Nov 4, 2020

msorens Nov 3, 2020

kvivek1115 Nov 4, 2020

msorens Nov 3, 2020

kvivek1115 Nov 4, 2020 •

edited

Loading

msorens Nov 4, 2020

msorens Nov 3, 2020

kvivek1115 Nov 4, 2020

msorens Nov 3, 2020

msorens Nov 3, 2020

kvivek1115 Nov 4, 2020

msorens Nov 3, 2020

msorens Nov 3, 2020

msorens Nov 3, 2020

kvivek1115 Nov 4, 2020

kvivek1115 commented Nov 9, 2020

		install_if_missing core/jq-static jq
		install_if_missing core/grpcurl grpcurl

	.PHONY: all static unit build compile proto test dep-ensure have-dep lint
	.PHONY: all static unit build test lint

		@@ -0,0 +1,5 @@
		# Sample data service

		Sample data service can be used to send scheduled data to Automate mimicing the data that would be send from a client.

	// Server is an sample-data server
	// Server is a sample-data server

	// NewServer returns an sample-data server
	// NewServer returns a sample-data server

sample-data-service mock infra client data #4398

sample-data-service mock infra client data #4398

Conversation

kvivek1115 commented Oct 12, 2020 • edited Loading

🔩 Description: What code changed, and why?

⛓️ Related Resources

👍 Definition of Done

👟 How to Build and Test the Change

✅ Checklist

📷 Screenshots, if applicable

netlify bot commented Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

kvivek1115 commented Oct 23, 2020

srenatus commented Oct 23, 2020

srenatus commented Oct 23, 2020

srenatus commented Oct 23, 2020

srenatus commented Oct 23, 2020

kvivek1115 commented Oct 23, 2020

Choose a reason for hiding this comment

lancewf left a comment

Choose a reason for hiding this comment

stevendanna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msorens Nov 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvivek1115 Nov 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvivek1115 commented Nov 9, 2020

kvivek1115 commented Oct 12, 2020 •

edited

Loading

netlify bot commented Oct 12, 2020 •

edited

Loading

msorens Nov 3, 2020 •

edited

Loading

kvivek1115 Nov 4, 2020 •

edited

Loading