Merge branch 'stable' into 'master'

* stable:
  (PDB-1810) Add anonymization and exportation for catalogs wire-v7
  (PDB-1810) Cleanup syntax in catalogs.clj for code-id patch
  (PDB-1810) Add documentation and non-nil test for code-id
  (PDB-1810) Add code-id field to catalogs

documentation/api/command/v1/commands.markdown

@@ -6,6 +6,7 @@ canonical: "/puppetdb/latest/api/command/v1/commands.html"
 
 [factsv4]: ../../wire_format/facts_format_v4.html
 [catalogv6]: ../../wire_format/catalog_format_v6.html
+[catalogv7]: ../../wire_format/catalog_format_v7.html
 [reportv5]: ../../wire_format/report_format_v5.html
 [reportv6]: ../../wire_format/report_format_v6.html
 [deactivatev3]: ../../wire_format/deactivate_node_format_v3.html
@@ -88,15 +89,12 @@ processed.
 
 ## List of Commands
 
-### "replace catalog", version 6
+### "replace catalog", version 7
 
-* All field names that were previously separated by dashes are
-  separated by underscores.
-
-* The catalog 'name' field has been renamed to 'certname'.
+* The nullable `code_id` property has been added.
 
 The payload is expected to be a Puppet catalog, as a JSON object,
-conforming exactly to the [catalog wire format v6][catalogv6]. Extra
+conforming exactly to the [catalog wire format v7][catalogv7]. Extra
 or missing fields are an error.
 
 ### "replace facts", version 4
@@ -133,6 +131,17 @@ Puppet resources. It is structured as a JSON object, conforming to the
 
 ## Deprecated Commands
 
+### "replace catalog", version 6
+
+* All field names that were previously separated by dashes are
+  separated by underscores.
+
+* The catalog 'name' field has been renamed to 'certname'.
+
+The payload is expected to be a Puppet catalog, as a JSON object,
+conforming exactly to the [catalog wire format v6][catalogv6]. Extra
+or missing fields are an error.
+
 ### "store report", version 5
 
 The version 5 store report command differs from version 4 in the addition of a

documentation/api/query/v4/catalogs.markdown

@@ -33,6 +33,7 @@ See [the Operators page.](./operators.html)
 * `version` (string): an arbitrary string that uniquely identifies each catalog for a node
 * `environment` (string): the environment associated with the catalog's certname
 * `transaction_uuid` (string): a string used to tie a catalog to a report from the same puppet run
+* `code_id` (string): a string used to tie a catalog to the Puppet code which generated the catalog
 * `hash` (string): sha1 hash of the resources of associated with a node's most
   recent catalog
 * `producer_timestamp` (string): a string representing the time at which the
@@ -51,6 +52,7 @@ the form:
       "environment" : <catalog environment>,
       "hash" : <sha1 sum of catalog resources>,
       "transaction_uuid" : <string to identify puppet run>,
+      "code_id" : <string to identify puppet code>,
       "producer_timestamp": <time of transmission by master>,
       "resources" : <expanded resources>,
       "edges" : <expanded edges>
@@ -102,6 +104,7 @@ This query will return the complete list of catalogs:
       "hash" : "62cdc40a78750144b1e1ee06638ac2dd0eeb9a46",
       "version" : "e4c339f",
       "transaction_uuid" : "53b72442-3b73-11e3-94a8-1b34ef7fdc95",
+      "code_id" : null,
       "producer_timestamp": "2014-10-13T20:46:00.000Z",
       "environment" : "production",
       "edges" : {...},
@@ -112,6 +115,7 @@ This query will return the complete list of catalogs:
       "hash" : "e1a4610ecbb3483fa5e637f42374b2cc46d06474",
       "version" : "449720",
       "transaction_uuid" : "9a3c8da6-f48c-4567-b24e-ddae5f80a6c6",
+      "code_id" : null,
       "producer_timestamp": "2014-11-20T02:15:20.861Z",
       "environment" : "production",
       "edges" : {...},
@@ -128,6 +132,7 @@ This query will return all catalogs with producer_timestamp after 2014-11-19:
       "hash" : "e1a4610ecbb3483fa5e637f42374b2cc46d06474",
       "version" : "449720",
       "transaction_uuid" : "9a3c8da6-f48c-4567-b24e-ddae5f80a6c6",
+      "code_id" : null,
       "producer_timestamp": "2014-11-20T02:15:20.861Z",
       "environment" : "production",
       "edges" : {...},
@@ -154,6 +159,7 @@ a JSON error message if the catalog is not found:
      "hash" : "62cdc40a78750144b1e1ee06638ac2dd0eeb9a46",
      "version" : "e4c339f",
      "transaction_uuid" : "53b72442-3b73-11e3-94a8-1b34ef7fdc95",
+     "code_id" : null,
      "producer_timestamp": "2014-10-13T20:46:00.000Z",
      "environment" : "production",
      "edges" : {...},

documentation/api/wire_format/catalog_format_v7.markdown

@@ -0,0 +1,213 @@
+---
+title: "PuppetDB 3.1 » API » Catalog Wire Format, Version 7"
+layout: default
+canonical: "/puppetdb/latest/api/wire_format/catalog_format_v7.html"
+---
+
+[containment]: /puppet/latest/reference/lang_containment.html
+[relationship]: /puppet/latest/reference/lang_relationships.html
+[chain]: /puppet/latest/reference/lang_relationships.html#chaining-arrows
+[metaparameters]: /puppet/latest/reference/lang_relationships.html#relationship-metaparameters
+[require]: /puppet/latest/reference/lang_relationships.html#the-require-function
+[resource_ref]: /puppet/latest/reference/lang_datatypes.html#resource-references
+[numbers]: /puppet/latest/reference/lang_datatypes.html#numbers
+[undef]: /puppet/latest/reference/lang_datatypes.html#undef
+[namevar]: /puppet/latest/reference/lang_resources.html#namenamevar
+[resource]: /puppet/latest/reference/lang_resources.html
+[title]: /puppet/latest/reference/lang_resources.html#title
+[type]: /puppet/latest/reference/lang_resources.html#type
+[attributes]: /puppet/latest/reference/lang_resources.html#attributes
+
+PuppetDB receives catalogs from puppet masters in the following wire format. This format is subtly different from the internal format used by Puppet so catalogs are converted by the [PuppetDB catalog terminus](../../connect_puppet_master.html) before they are sent.
+
+Catalog Interchange Format
+-----
+
+### Version
+
+This is **version 7** of the catalog interchange format.
+
+
+### Encoding
+
+The entire catalog is serialized as JSON, which requires strict UTF-8 encoding. Unless otherwise noted, null is not allowed anywhere in the catalog.
+
+### Main Data Type: Catalog
+
+     {
+      "certname": <string>,
+      "version": <string>,
+      "environment": <string>,
+      "transaction_uuid": <string>,
+      "code_id": <string>,
+      "producer_timestamp": <datetime>,
+      "edges":
+          [<edge>, <edge>, ...],
+      "resources":
+          [<resource>, <resource>, ...]
+     }
+
+#### `certname`
+
+String. The name of the node for which the catalog was compiled.
+
+#### `version`
+
+String. An arbitrary string that uniquely identifies this specific catalog across time for a single node. This is controlled by Puppet's [`config_version` setting](/references/latest/configuration.html#configversion) and is usually the seconds elapsed since the epoch.
+
+#### `environment`
+
+String. The environment associated to the node when the catalog was compiled.
+
+#### `edges`
+
+List of [`<edge>` objects](#data-type-edge). **Every** [relationship][] between any two resources in the catalog, which may have been made with [chaining arrows][chain], [metaparameters][], or [the `require` function][require].
+
+  > **Notes:**
+  >
+  > * "Autorequire" relationships are not currently encoded in the catalog.
+  > * This key is significantly different from its equivalent in Puppet's internal catalog format, which only encodes containment edges.
+
+#### `resources`
+
+List of [`<resource>` objects](#data-type-resource). Contains **every** resource in the catalog.
+
+#### `transaction_uuid`
+
+String. A string used to match the catalog with the corresponding report that was issued during the same puppet run.
+This field may be `null`.
+
+#### `code_id`
+
+String. A string used to match the catalog with the Puppet code which generated the catalog.
+This field may be `null`.
+
+#### `producer_timestamp`
+
+Datetime.  The time of catalog submission from the master to PuppetDB.  This
+field is currently populated by the master.
+
+### Data Type: `<string>`
+
+A JSON string. Because the catalog is UTF-8, these must also be UTF-8.
+
+### Data Type: `<integer>`
+
+A JSON int.
+
+### Data Type: `<boolean>`
+
+A JSON boolean.
+
+### Data Type: `<datetime>`
+A JSON string representing a date and time (with time zone), formatted based on
+the recommendations in ISO8601; so, e.g., for a UTC time, the String would be
+formatted as `YYYY-MM-DDThh:mm:ss.sssZ`.  For non-UTC, the `Z` may be replaced
+with `±hh:mm` to represent the specific timezone.
+
+### Data Type: `<edge>`
+
+A JSON object of the following form, which represents a [relationship][] between two resources:
+
+    {"source": <resource-spec>,
+     "target": <resource-spec>,
+     "relationship": <relationship>}
+
+All edges are normalized so that the "source" resource is managed **before** the "target" resource. To do this, the Puppet language's "require" and "subscribe" [relationship types][relationship] are munged into "required-by" and "subscription-of" when they are converted into edges.
+
+The keys of an edge are `source`, `target`, and `relationship`, all of which are required.
+
+#### `source`
+
+A [`<resource-spec>`](#data-type-resource-spec). The resource which should be managed **first.**
+
+#### `target`
+
+A [`<resource-spec>`](#data-type-resource-spec). The resource which should be managed **second.**
+
+#### `relationship`
+
+A [`<relationship>`](#data-type-relationship). The way the two resources are related.
+
+### Data Type: `<resource-spec>`
+
+(Synonym: `<resource-hash>`.)
+
+The JSON representation of a [resource reference][resource_ref] (single-resource kind). An object of the following form:
+
+    {"type": <string>,
+     "title": <string>}
+
+The resource named by a resource-spec **must** exist in the catalog's `"resources"` list. Note also that the title must be the resource's actual [title][], rather than an alias or [name/namevar][namevar].
+
+### Data Type: `<relationship>`
+
+One of the following exact strings, when used in the `relationship` key of an [`<edge>` object](#data-type-edge):
+
+* `contains`
+* `before`
+* `required-by`
+* `notifies`
+* `subscription-of`
+
+**Note:** Regardless of the relationship type, the "source" resource is always managed **before** the "target" resource. This means that, functionally speaking, `required-by` is a synonym of `before` and `subscription-of` is a synonym of `notifies`. In this catalog format, the different relationship types preserve information about the _origin_ of the relationship.
+
+
+String            | Relationship Type        | Origin of Relationship
+------------------|--------------------------|-----------------------
+`contains`        | [containment][]          | Class or defined type [containment][]
+`before`          | ordering                 | `before` metaparam on source, or `->` chaining arrow
+`required-by`     | ordering                 | `require` metaparam on target, or `require` function
+`notifies`        | ordering w/ notification | `notify` metaparam on source, or `~>` chaining arrow
+`subscription-of` | ordering w/ notification | `subscribe` metaparam on target
+
+
+### Data Type: `<resource>`
+
+A JSON object of the following form, which represents a [Puppet resource][resource]:
+
+    {"type": <string>,
+     "title": <string>,
+     "aliases": [<string>, <string>, ...],
+     "exported": <boolean>,
+     "file": <string>,
+     "line": <string>,
+     "tags": [<string>, <string>, ...],
+     "parameters": {<string>: <JSON object>,
+                    <string>: <JSON object>,
+                    ...}
+    }
+
+The eight keys in a resource object are `type`, `title`, `aliases`, `exported`, `file`, `line`, `tags` and `parameters`. All of them are **required.**
+
+#### `type`
+
+String. The [type][] of the resource, **capitalized.** (E.g. `File`, `Service`, `Class`, `Apache::Vhost`.) Note that every segment must be capitalized if the type includes a namespace separator (`::`).
+
+#### `title`
+
+String. The [title][] of the resource.
+
+#### `aliases`
+
+List of strings. Includes **every** alias for the resource, including the value of its [name/namevar][namevar] and any extra names added with the `"alias"` metaparameter.
+
+#### `exported`
+
+Boolean. Whether or not this is an exported resource.
+
+#### `file`
+
+String. The manifest file in which the resource definition is located.
+
+#### `line`
+
+Positive integer. The line (of the containing manifest file) at which the resource definition can be found.
+
+#### `tags`
+
+List of strings. Includes every tag the resource has. This is a normalized superset of the value of the resource's `tag` attribute.
+
+#### `parameters`
+
+JSON object. Includes all of the resource's [attributes][] and their associated values. The value of an attribute may be any JSON data type, but Puppet will only provide booleans, strings, arrays, and hashes --- [resource references][resource_ref] and [numbers][] in attributes are converted to strings before being inserted into the catalog. Attributes with [undef][] values are not added to the catalog.

puppet/lib/puppet/indirector/catalog/puppetdb.rb

@@ -10,7 +10,7 @@ class Puppet::Resource::Catalog::Puppetdb < Puppet::Indirector::REST
   def save(request)
     profile("catalog#save", [:puppetdb, :catalog, :save, request.key]) do
       catalog = munge_catalog(request.instance, extract_extra_request_data(request))
-      submit_command(request.key, catalog, CommandReplaceCatalog, 6)
+      submit_command(request.key, catalog, CommandReplaceCatalog, 7)
     end
   end
 
@@ -24,6 +24,7 @@ def extract_extra_request_data(request)
       :transaction_uuid => request.options[:transaction_uuid],
       :environment => request.environment.to_s,
       :producer_timestamp => request.options[:producer_timestamp] || Time.now.iso8601(5),
+      :code_id => request.options[:code_id],
     }
   end
 
@@ -51,6 +52,7 @@ def munge_catalog(catalog, extra_request_data = {})
       add_environment(data, extra_request_data[:environment])
       add_producer_timestamp(data, extra_request_data[:producer_timestamp])
       change_name_to_certname(data)
+      add_code_id(data, extra_request_data[:code_id])
 
       data
     end
@@ -114,6 +116,18 @@ def add_transaction_uuid(hash, transaction_uuid)
     hash
   end
 
+  # Include code_id in hash, returning the complete hash.
+  #
+  # @param hash [Hash] original data hash
+  # @param code_id [String] code_id
+  # @return [Hash] returns original hash augmented with transaction_uuid
+  # @api private
+  def add_code_id(hash, code_id)
+    hash['code_id'] = code_id
+
+    hash
+  end
+
   # Version is an integer (time since epoch in millis). The wire
   # format specifies version should be a string
   #

puppet/spec/unit/indirector/catalog/puppetdb_spec.rb

@@ -23,6 +23,7 @@
       cat
     end
     let(:options) {{
+      :code_id => 'my_git_sha1',
       :transaction_uuid => 'abcdefg',
       :environment => 'my_environment',
       :producer_timestamp => "a test",
@@ -41,7 +42,7 @@ def save
       command_payload = subject.munge_catalog(catalog, options)
       payload = {
         :command => Puppet::Util::Puppetdb::CommandNames::CommandReplaceCatalog,
-        :version => 6,
+        :version => 7,
         :payload => command_payload,
       }.to_json
 
@@ -695,7 +696,7 @@ def test_file_require(resource_title, require_title)
         result = subject.munge_catalog(catalog)
 
         result.keys.should =~ ['certname', 'version', 'edges', 'resources',
-          'transaction_uuid', 'environment', 'producer_timestamp']
+          'transaction_uuid', 'environment', 'producer_timestamp', "code_id"]
       end
     end
   end

src/puppetlabs/puppetdb/anonymizer.clj

@@ -312,6 +312,7 @@
     (-> resource
         (utils/update-when ["file"] anonymize-leaf :file newcontext config)
         (utils/update-when ["line"] anonymize-leaf :line newcontext config)
+        (utils/update-when ["code_id"] anonymize-leaf :code_id newcontext config)
         (update "parameters" anonymize-parameters newcontext config)
         (update "tags" anonymize-tags newcontext config)
         (update "title" anonymize-leaf :title newcontext config)