(maint) Merge 2.3.1 into master (fact paths, etc.) #1331

rbrw · 2015-03-31T18:57:01Z

    (maint) Merge 2.3.1 into master (fact paths, etc.)

    * stable:
      (PDB-1328) Add contributors to and edit release notes
      (PDB-1328) Add preliminary 2.3.1 release notes
      (PDB-1031) Verify gc behavior in update race test
      (PDB-1031) Reproduce the fact paths constraint issue
      (PDB-1031) Speed migration with unique facts table
      (PDB-1031) Accommodate move of fact paths to facts
      (PDB-1031) Fix query engine, v3 endpoints for fact_path move
      (docs) Replace Checkin with Check In
      (PDB-1342) Fix trusted facts fallback
      (docs) Add .html suffixes to 2.3.0 release notes
      (PDB-1307) Update to tk-jetty 1.3.0
      (maint) applied-migrations: don't ignore exceptions

 documentation/release_notes.markdown                   |  87 +++++-
 documentation/repl.markdown                            |   3 +-
 documentation/scaling_recommendations.markdown         |   2 +-
 project.clj                                            |   2 +-
 puppet/lib/puppet/indirector/facts/puppetdb.rb         |   2 +-
 puppet/spec/unit/indirector/facts/puppetdb_spec.rb     |  25 +-
 src/puppetlabs/puppetdb/cli/services.clj               |   5 +-
 src/puppetlabs/puppetdb/facts.clj                      |  90 +++---
 src/puppetlabs/puppetdb/query.clj                      |   2 +-
 src/puppetlabs/puppetdb/query_eng/engine.clj           |  22 +-
 src/puppetlabs/puppetdb/scf/migrate.clj                | 129 +++++++--
 src/puppetlabs/puppetdb/scf/migration_legacy.clj       | 293 +++++++++++++++++++
 src/puppetlabs/puppetdb/scf/storage.clj                | 576 ++++++++++++++++++-------------------
 test/puppetlabs/puppetdb/command_test.clj              | 155 +++++++++-
 test/puppetlabs/puppetdb/facts_test.clj                |  72 -----
 test/puppetlabs/puppetdb/scf/migrate_test.clj          | 100 ++++++-
 test/puppetlabs/puppetdb/scf/migration_legacy_test.clj |  76 +++++
 test/puppetlabs/puppetdb/scf/storage_test.clj          | 157 ++++++++--
 18 files changed, 1303 insertions(+), 495 deletions(-)

Check to see if there is a schema_migrations table before querying it. This eliminates these error messages from the postgresql log: STATEMENT: SELECT version FROM schema_migrations ORDER BY version ERROR: relation "schema_migrations" does not exist at character 21 A very large number of these messages are produced during the unit tests.

The 2.3.0 release notes were missing .html extensions to some of the links. This patch remedies that.

When adding trusted facts, a fallback mechanism is used to pull them from the node object if there aren’t any there already. This code is broken in puppet 4; it’s unclear whether it happened to work in puppet 3 because there was a global request object available or because this code was never called. In any case, the change works in both.

…update-tk-jetty (PDB-1307) Update to tk-jetty 1.3.0

…d-migrations-without-ignoring-exceptions (maint) applied-migrations: don't ignore exceptions

…fix-trusted-node-data (PDB-1342) Fix trusted facts fallback

* checkin-typo: (docs) Replace Checkin with Check In

Change the query engine and the v3 query endpoints to handle the migration of fact_paths from fact_values to facts. cf. (PDB-1225) [rlb@puppetlabs.com: remove incidental whitespace; adjust commit message]

Now that facts refer directly to paths: (factset_id fact_path_id fact_value_id) i.e. the path_id is no longer linked to a fact_value, and the fact_path_id column has been dropped from fact_values, and the value_type_id has been dropped from fact_paths, adjust the code to accommodate those changes. Note that the new migration-baggage namespace contains code effectively just copy-pasted from the old storage.clj (plus or minus some minor edits). This includes migration 28. cf. (PDB-1224) (PDB-1225) (PDB-1226)

Copy just (factset_id, path, hash) to a temporary table and then use that to rebuild facts once we've eliminated the duplicates from fact_paths and fact_values. In tests, this improved performance substantially. cf. (PDB-1224) [rlb@puppetlabs.com: adjust commit message; adjust for migration updates]

This patch creates a unit test that recreates the following error: update or delete on table "fact_paths" violates foreign key constraint "fact_values_path_id_fk" on table "fact_values" [rlb@puppetlabs.com: remove demonstration output; instrument delete-pending-path-id-orphans! instead of delete-facts!, which is no longer called by replace-facts!; remove unused with-pg-test-db]

…lobal-inc-gc (PDB-1031) Lift fact paths into facts (also PDB-1224, PDB-1225, PDB-1226, PDB-1227)

This commit adds recent contributors to the release notes for 2.3.1. This commit also edits the description of the fact_paths issue to make the issue more clear as well as updating the ticket links/numbers.

…328-2.3.1-release-notes Ticket/stable/pdb 1328 2.3.1 release notes

* stable: (PDB-1328) Add contributors to and edit release notes (PDB-1328) Add preliminary 2.3.1 release notes (PDB-1031) Verify gc behavior in update race test (PDB-1031) Reproduce the fact paths constraint issue (PDB-1031) Speed migration with unique facts table (PDB-1031) Accommodate move of fact paths to facts (PDB-1031) Fix query engine, v3 endpoints for fact_path move (docs) Replace Checkin with Check In (PDB-1342) Fix trusted facts fallback (docs) Add .html suffixes to 2.3.0 release notes (PDB-1307) Update to tk-jetty 1.3.0 (maint) applied-migrations: don't ignore exceptions

pljenkinsro · 2015-03-31T20:10:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/960/

wkalt · 2015-03-31T20:12:19Z

documentation/release_notes.markdown

@@ -302,6 +302,73 @@ derive data from a PuppetDB export tarball.

 TODO


@rlb does this need to go?

@wkalt perhaps not, that's a placeholder (already) in master, though I think we may eventually want to move it back up near the top where we normally put the contributors?

leave it, pre-existing & out of scope.

wkalt · 2015-03-31T20:16:19Z

this looks good to me on first glance

…hs-etc (maint) Merge 2.3.1 into master (fact paths, etc.)

rbrw and others added 19 commits March 24, 2015 12:38

(PDB-1307) Update to tk-jetty 1.3.0

9c59032

(docs) Add .html suffixes to 2.3.0 release notes

113206f

The 2.3.0 release notes were missing .html extensions to some of the links. This patch remedies that.

Merge pull request puppetlabs#1327 from mullr/ticket/stable/pdb-1307-…

ec9aec8

…update-tk-jetty (PDB-1307) Update to tk-jetty 1.3.0

Merge pull request puppetlabs#1320 from rbrw/ticket/stable/get-applie…

11e6511

…d-migrations-without-ignoring-exceptions (maint) applied-migrations: don't ignore exceptions

Merge pull request puppetlabs#1326 from mullr/ticket/stable/pdb-1342-…

7407872

…fix-trusted-node-data (PDB-1342) Fix trusted facts fallback

(docs) Replace Checkin with Check In

6804e75

Merge branch 'checkin-typo' into stable

f570d3d

* checkin-typo: (docs) Replace Checkin with Check In

(PDB-1031) Fix query engine, v3 endpoints for fact_path move

dbb9f71

Change the query engine and the v3 query endpoints to handle the migration of fact_paths from fact_values to facts. cf. (PDB-1225) [rlb@puppetlabs.com: remove incidental whitespace; adjust commit message]

(PDB-1031) Verify gc behavior in update race test

dd21714

Merge pull request puppetlabs#1296 from rbrw/ticket/stable/pdb-1226-g…

dda38b6

…lobal-inc-gc (PDB-1031) Lift fact paths into facts (also PDB-1224, PDB-1225, PDB-1226, PDB-1227)

(PDB-1328) Add preliminary 2.3.1 release notes

a97ecf2

(PDB-1328) Add contributors to and edit release notes

79fab2f

This commit adds recent contributors to the release notes for 2.3.1. This commit also edits the description of the fact_paths issue to make the issue more clear as well as updating the ticket links/numbers.

Merge pull request puppetlabs#1330 from ajroetker/ticket/stable/pdb-1…

63660a2

…328-2.3.1-release-notes Ticket/stable/pdb 1328 2.3.1 release notes

rbrw added the work in progress (...and please don't merge) label Mar 31, 2015

wkalt reviewed Mar 31, 2015
View reviewed changes

kbarber added a commit that referenced this pull request Mar 31, 2015

Merge pull request #1331 from rbrw/ticket/master/merge-2.3.1-fact-pat…

e571052

…hs-etc (maint) Merge 2.3.1 into master (fact paths, etc.)

kbarber merged commit e571052 into puppetlabs:master Mar 31, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(maint) Merge 2.3.1 into master (fact paths, etc.) #1331

(maint) Merge 2.3.1 into master (fact paths, etc.) #1331

rbrw commented Mar 31, 2015

pljenkinsro commented Mar 31, 2015

wkalt Mar 31, 2015

rbrw Mar 31, 2015

kbarber Mar 31, 2015

wkalt commented Mar 31, 2015

		@@ -302,6 +302,73 @@ derive data from a PuppetDB export tarball.

		TODO

(maint) Merge 2.3.1 into master (fact paths, etc.) #1331

(maint) Merge 2.3.1 into master (fact paths, etc.) #1331

Conversation

rbrw commented Mar 31, 2015

pljenkinsro commented Mar 31, 2015

wkalt Mar 31, 2015

Choose a reason for hiding this comment

rbrw Mar 31, 2015

Choose a reason for hiding this comment

kbarber Mar 31, 2015

Choose a reason for hiding this comment

wkalt commented Mar 31, 2015