Spark implementation of Materials API

[akshaydewan]

- Get materials moved from /go/api/config/materials to /go/api/materials, because it lists all materials, including the ones from config repo. (The documentation would have you believe it just lists the ones from cruise-config.xml)
Edit: Kept the endpoint /go/api/config/materials, because we also have material instances.

• Moved PaginationRepresenter to api-base
• Notify urls have been changed changed to /go/api/admin/materials/[scm_type]/notify instead of /go/api/admin/material/notify/[scm_type] to make it consistent with webooks.
• Using the plural /materials everywhere to make it consistent with other endpoints implemented in spark
• /notify payload and response changed to json instead of POST data. The JSON payload is of the format {"repository_url": "foo"}
• The Notify endpoint for svn was earlier using UUID as a parameter. This parameter has been changed to repository_url to make it consistent with git and hg
• All the material properties will be shown (similar to pipeline config API) in the GET endpoint for materials

[bdpiprava]

Mostly looks okay to me.

Gist of changes

Only admins are allowed to force MDU on material

curl "http://localhost:8153/go/api/admin/materials/notify/svn" \
	-u "admin:badger" \
	-H "Accept: application/vnd.go.cd.v1+json" \
	-X POST -d "{\"uuid\": \"some-id\"}"

Users are allowed to get material configs for the pipelines which they have access to

curl 'http://localhost:8153/go/api/materials' \
    -H "Accept: application/vnd.go.cd.v1+json" \
    -u 'admin:badger'

Users are allowed to get material modifications for the pipelines which they have access to

curl 'http://localhost:8153/go/api/materials/50660e7819...eb209/modifications' \
    -H "Accept: application/vnd.go.cd.v1+json" \
    -u 'admin:badger'

• API documentation needs to be updated with new URLs, payload and version header.

---

PS: There is an inconsistency between the APIs to represent the material type in the payload.

e.g.

Materials API response where type=Git

curl 'http://localhost:8153/go/api/materials' \
    -H "Accept: application/vnd.go.cd.v1+json" \
    -u 'admin:badger'

# Response
{
  "_links" : {
    "doc" : {
      "href" : "https://api.gocd.org/19.5.0/#materials"
    },
    "self" : {
      "href" : "http://localhost:8153/go/api/materials"
    }
  },
  "_embedded" : {
    "materials" : [ {
      "fingerprint" : "50660e7819...ceb209",
      "type" : "Git",
      "description" : "URL: https://github.com/bdpiparva/sample_repo, Branch: master"
    } ]
  }
}

Pipeline config response where type=git

curl 'http://localhost:8153/go/api/admin/pipelines/up42' \
      -u 'admin:badger' \
      -H 'Accept: application/vnd.go.cd.v7+json'

# Response
{
  "_links" : {
    ...
  },
  ...
  "materials" : [ {
    "type" : "git",
    "attributes" : {
      "url" : "https://github.com/bdpiparva/sample_repo",
      "destination" : null,
      "filter" : null,
      "invert_filter" : false,
      "name" : null,
      "auto_update" : true,
      "branch" : "master",
      "submodule_folder" : null,
      "shallow_clone" : false
    }
  } ],
  ...
}

Should we address it or not?

[bdpiprava]

Related issue: #3924

[akshaydewan]

Further changes done on this API:

GET materials

curl -u admin:badger -H Accept:application/vnd.go.cd.v1+json http://localhost:8153/go/api/config/materials         

{
  "_links" : {
    "doc" : {
      "href" : "https://api.gocd.org/19.5.0/#materials"
    },
    "self" : {
      "href" : "http://localhost:8153/go/api/config/materials"
    }
  },
  "_embedded" : {
    "materials" : [ {
      "type" : "git",
      "fingerprint" : "458ce82419bdbe41a4007ad713a730e5e467ed4fbefb421bf58aeff3e9c36e4b",
      "attributes" : {
        "url" : "https://github.com/akshaydewan/arduino",
        "destination" : null,
        "filter" : null,
        "invert_filter" : false,
        "name" : null,
        "auto_update" : false,
        "branch" : "master",
        "submodule_folder" : null,
        "shallow_clone" : false
      }
    }, {
      "type" : "svn",
      "fingerprint" : "6b0cd6b9181434866c555f3c3bb780e950389a456764c876d804c848efbad554",
      "attributes" : {
        "url" : "https://github.com/akshaydewan/arduino",
        "destination" : "ads",
        "filter" : null,
        "invert_filter" : false,
        "name" : null,
        "auto_update" : false,
        "username" : "admin",
        "encrypted_password" : "AES:lTFKwi5NvrlqmQ3LOQ5UQA==:Ebggz5N27w54NrhSXKIbng==",
        "check_externals" : false
      }
    }, {
      "type" : "hg",
      "fingerprint" : "f6b61bc6b33e524c2a94c5be4a4661e5f1d2b74ca089418de20de10b282e39e9",
      "attributes" : {
        "url" : "ssh://hg@bitbucket.org/akshaydewan86/gocd-ssh-hg",
        "destination" : "sadasd",
        "filter" : null,
        "invert_filter" : false,
        "name" : null,
        "auto_update" : false,
        "branch" : ""
      }
    }, {
      "type" : "dependency",
      "fingerprint" : "21f7963a8eca6cced5b4f20b57b4e9cb8edaed29d683c83621a77dfb499c553d",
      "attributes" : {
        "pipeline" : "up42",
        "stage" : "up42_stage",
        "name" : "up42_material",
        "auto_update" : true
      }
    } ]
  }
}

POST to notify a material

curl -u admin:badger -H Content-Type:application/json -H Accept:application/vnd.go.cd.v1+json http://localhost:8153/go/api/admin/materials/hg/notify -d '{"repository_url": "ssh://hg@bitbucket.org/akshaydewan86/gocd-ssh-hg"}'

{
  "message" : "The material is now scheduled for an update. Please check relevant pipeline(s) for status."
}

GET material modifications

curl -u admin:badger -H Accept:application/vnd.go.cd.v1+json http://localhost:8153/go/api/materials/458ce82419bdbe41a4007ad713a730e5e467ed4fbefb421bf58aeff3e9c36e4b/modifications

{
  "_links" : {
    "self" : {
      "href" : "http://localhost:8153/go/api/materials/458ce82419bdbe41a4007ad713a730e5e467ed4fbefb421bf58aeff3e9c36e4b/modifications"
    },
    "doc" : {
      "href" : "https://api.gocd.org/19.5.0/#materials"
    },
    "find" : {
      "href" : "http://localhost:8153/go/api/materials/:fingerprint/modifications/{offset}"
    }
  },
  "_embedded" : {
    "modifications" : [ {
      "id" : 2,
      "user_name" : "Akshay Dewan <akudewan@gmail.com>",
      "revision" : "92385c780e7d89db9a3aaeae573b1756fbe6c6b2",
      "modified_time" : 1404828986000,
      "comment" : "Adding visualizer and knock sensor"
    } ],
    "pagination" : {
      "page_size" : 10,
      "offset" : 0,
      "total" : 1
    }
  }
}

I observed that the Modifications endpoint returns a server error if the pipeline hasn't been run yet. This is because no MaterialInstance has been created. This will need to be handled better.

[adityasood]

The PR was validated on 19.6.0-9340