(PDB-1072) Add endpoint to mirror report command structure. #1193

wkalt · 2014-12-26T21:33:36Z

This commit adds an endpoint called event-reports, the return value of which
can be easily munged for resubmission as a report. In an HA context, this will
allow us to transfer reports from one database to another over HTTP without
needing a second query for the events associated with a report.

wkalt · 2014-12-26T21:34:25Z

open to suggestions on renaming this endpoint

wkalt · 2014-12-26T21:35:39Z

src/puppetlabs/puppetdb/jdbc.clj

+                 hash, puppet_version, receive_time, report_format,
+                 start_time, end_time, transaction_uuid, status, environment,
+                 configuration_version, certname FROM (%s) distinct_names %s%s%s) %s"
+                 sql sql inner-order-by limit-clause offset-clause order-by-clause)


need to think about whether we have better options than this pattern. The issue is limit/offset on API responses that comprise multiple rows.

pljenkinsro · 2014-12-26T22:30:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/321/

pljenkinsro · 2014-12-27T00:16:21Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/322/

pljenkinsro · 2014-12-27T02:24:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/324/

pljenkinsro · 2014-12-27T03:23:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/325/

ajroetker · 2015-01-05T18:40:36Z

src/puppetlabs/puppetdb/http/event_reports.clj

+            [puppetlabs.puppetdb.middleware :refer [verify-accepts-json validate-query-params
+                                                    wrap-with-paging-options]]
+            [puppetlabs.puppetdb.jdbc :as jdbc]
+            [puppetlabs.puppetdb.http :as http]))


Are all these dependencies necessary?

kbarber · 2015-01-05T19:34:20Z

So here is the thing, this new endpoint looks almost exactly like /reports ... but without the downstream information like events ... I'm wondering if we should just call this /reports instead, and let people use the extract field selector capability if they want the old report format? That would save us from having to create a brand new end-point.

wkalt · 2015-01-05T19:40:23Z

that's fine with me in concept, but we'll have to extend extract to cover non-queryable fields, since the event-reports field of this endpoint is not queryable. Not sure what that will take; I'll give it some thought.

wkalt · 2015-01-05T22:26:56Z

actually, that's not going to be an issue in the slightest. Happy to take this route.

pljenkinsro · 2015-01-06T00:53:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/346/

puppetcla · 2015-01-06T18:00:28Z

CLA signed by all contributors.

pljenkinsro · 2015-01-07T02:26:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/360/

pljenkinsro · 2015-01-07T03:25:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/361/

pljenkinsro · 2015-01-07T03:26:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/362/

pljenkinsro · 2015-01-07T04:40:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/363/

pljenkinsro · 2015-01-07T04:50:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/364/

wkalt · 2015-01-07T16:37:29Z

@pljenkinsro retest this please

pljenkinsro · 2015-01-07T16:40:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/365/

kbarber · 2015-01-07T16:44:13Z

@pljenkinsro retest this please

pljenkinsro · 2015-01-07T18:04:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/367/

pljenkinsro · 2015-01-07T20:24:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/373/

wkalt · 2015-01-07T20:25:27Z

@pljenkinsro retest this please

pljenkinsro · 2015-01-07T20:54:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/374/

wkalt · 2015-01-07T20:56:25Z

@pljenkinsro retest this please

pljenkinsro · 2015-01-07T21:01:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/375/

wkalt · 2015-01-07T21:06:31Z

@pljenkinsro retest this please

pljenkinsro · 2015-01-07T22:07:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/376/

pljenkinsro · 2015-01-07T22:18:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/378/

pljenkinsro · 2015-01-07T22:37:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/379/

wkalt · 2015-01-07T23:36:16Z

@pljenkinsro retest this please

pljenkinsro · 2015-01-08T01:04:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/381/

senior · 2015-01-08T03:18:44Z

test/puppetlabs/puppetdb/http/reports_test.clj


-(defn remove-receive-times
+(defn treat-reports-response


Can you come up with a more descriptive function name here? Maybe munge-for-compare, or prep-for-compare or something like that?

This commit changes the /v4/reports endpoint to mirror the store-report command structure -- this amounts to an additional field containing resource-events. The upshot of this is that the api response can be easily munged for resubmission as a report. In an HA context, this will allow us to transfer reports from one database to another over HTTP without needing a second query for the events associated with a report.

pljenkinsro · 2015-01-08T15:56:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/383/

wkalt · 2015-01-08T16:00:12Z

@pljenkinsro retest this please

pljenkinsro · 2015-01-08T17:02:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/384/

senior · 2015-01-08T18:48:53Z

+1 from me

…reports (PDB-1072) Add endpoint to mirror report command structure.

wkalt reviewed Dec 26, 2014
View reviewed changes

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from e8bf0b5 to 019ec8b Compare December 26, 2014 23:12

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch 2 times, most recently from 2518125 to 6dedcea Compare December 27, 2014 01:16

ajroetker reviewed Jan 5, 2015
View reviewed changes

kbarber added the work in progress (...and please don't merge) label Jan 5, 2015

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from 6dedcea to fd3602d Compare January 5, 2015 22:22

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch 2 times, most recently from 57972b7 to f8785fe Compare January 7, 2015 02:26

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch 3 times, most recently from fe7a9aa to 1f15228 Compare January 7, 2015 02:34

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from 1f15228 to 969f545 Compare January 7, 2015 04:35

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from 969f545 to 85acb27 Compare January 7, 2015 04:47

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from 85acb27 to e1479b9 Compare January 7, 2015 19:28

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from e1479b9 to 3ffb104 Compare January 7, 2015 22:10

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from 3ffb104 to 973e08f Compare January 7, 2015 22:32

wkalt removed the work in progress (...and please don't merge) label Jan 7, 2015

senior reviewed Jan 8, 2015
View reviewed changes

wkalt force-pushed the ticket/master/pdb-1072-symmetric-reports branch from 973e08f to 5b015a2 Compare January 8, 2015 15:53

senior added a commit that referenced this pull request Jan 8, 2015

Merge pull request #1193 from wkalt/ticket/master/pdb-1072-symmetric-…

09cc3ef

…reports (PDB-1072) Add endpoint to mirror report command structure.

senior merged commit 09cc3ef into puppetlabs:master Jan 8, 2015


		(defn remove-receive-times
		(defn treat-reports-response

(PDB-1072) Add endpoint to mirror report command structure. #1193

(PDB-1072) Add endpoint to mirror report command structure. #1193

Conversation

wkalt commented Dec 26, 2014

wkalt commented Dec 26, 2014

wkalt Dec 26, 2014

Choose a reason for hiding this comment

pljenkinsro commented Dec 26, 2014

pljenkinsro commented Dec 27, 2014

pljenkinsro commented Dec 27, 2014

pljenkinsro commented Dec 27, 2014

ajroetker Jan 5, 2015

Choose a reason for hiding this comment

kbarber commented Jan 5, 2015

wkalt commented Jan 5, 2015

wkalt commented Jan 5, 2015

pljenkinsro commented Jan 6, 2015

puppetcla commented Jan 6, 2015

pljenkinsro commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

wkalt commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

kbarber commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

wkalt commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

wkalt commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

wkalt commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

pljenkinsro commented Jan 7, 2015

wkalt commented Jan 7, 2015

pljenkinsro commented Jan 8, 2015

senior Jan 8, 2015

Choose a reason for hiding this comment

pljenkinsro commented Jan 8, 2015

wkalt commented Jan 8, 2015

pljenkinsro commented Jan 8, 2015

senior commented Jan 8, 2015