Add AMI set entity and its create and delete endpoints #85

Merged
merged 4 commits into from Jul 4, 2016

Conversation

Projects
None yet
3 participants
@anarute
Contributor

anarute commented Jun 6, 2016

I was not sure about its endpoints, if I am using the correct AMI Set properties.

src/ami-set.js
+
+ properties: {
+
+ amiSetId: base.Entity.types.String,

This comment has been minimized.

@jhford

jhford Jun 6, 2016

Collaborator

since we're in the AmiSet class, let's call this just 'id'

@jhford

jhford Jun 6, 2016

Collaborator

since we're in the AmiSet class, let's call this just 'id'

src/ami-set.js
+ properties: {
+
+ amiSetId: base.Entity.types.String,
+ amis: base.Entity.types.JSON,

This comment has been minimized.

@jhford

jhford Jun 6, 2016

Collaborator

Can you write a comment here that describes what this object should look like? It's really helpful to know what to expect in the data when coding against this Entity in the future.

@jhford

jhford Jun 6, 2016

Collaborator

Can you write a comment here that describes what this object should look like? It's really helpful to know what to expect in the data when coding against this Entity in the future.

src/api-v1.js
+ deferAuth: true,
+ scopes: [['aws-provisioner:manage-ami-set:<amiSetId>']],
+ input: 'create-ami-set-request.json#',
+ output: 'get-ami-set-response.json#',

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

These (input and output) point to the schemas we talked about today. You can just omit these lines for the moment, and then return to add schemas in a later pull request.

@djmitche

djmitche Jun 6, 2016

Contributor

These (input and output) point to the schemas we talked about today. You can just omit these lines for the moment, and then return to add schemas in a later pull request.

src/api-v1.js
+ let input = req.body;
+ let amiSet = req.params.amiSetId;
+
+ input.lastModified = new Date();

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

A lastModified property is a good idea, but you'll need to include it in src/ami-set.js as well.

@djmitche

djmitche Jun 6, 2016

Contributor

A lastModified property is a good idea, but you'll need to include it in src/ami-set.js as well.

src/api-v1.js
+ // Publish pulse message
+ await this.publisher.amiSetCreated({
+ amiSet: amiSetId,
+ });

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

I don't think we need to send a pulse message on AMI set creation. It doesn't hurt, and we could add it back in later.

For WorkerTypes, this message is required so that the provisioner backend can look at the new workerType and start creating EC2 instance for it. For new AmiSets, there's no need for the backend to do anything.

@djmitche

djmitche Jun 6, 2016

Contributor

I don't think we need to send a pulse message on AMI set creation. It doesn't hurt, and we could add it back in later.

For WorkerTypes, this message is required so that the provisioner backend can look at the new workerType and start creating EC2 instance for it. For new AmiSets, there's no need for the backend to do anything.

This comment has been minimized.

@jhford

jhford Jun 6, 2016

Collaborator

We don't actually consume any pulse messages in the provisioner. We scan the worker type table on each iteration, since we need to load all the worker types anyway to see what the max/min capacity, scaling ratio and instance types. We could maintain that state in the backend process, but that's not there yet and would be a fairly substantial code change. I don't know if the Azure overhead (tech or billing wise) is worth the trouble.

That said, I don't think that we really need to have pulse messages for AmiSets. The important Pulse messages are for WorkerType updates, since those are (well, should be) used to gracefully shutdown old instances when we update the worker type definition.

@jhford

jhford Jun 6, 2016

Collaborator

We don't actually consume any pulse messages in the provisioner. We scan the worker type table on each iteration, since we need to load all the worker types anyway to see what the max/min capacity, scaling ratio and instance types. We could maintain that state in the backend process, but that's not there yet and would be a fairly substantial code change. I don't know if the Azure overhead (tech or billing wise) is worth the trouble.

That said, I don't think that we really need to have pulse messages for AmiSets. The important Pulse messages are for WorkerType updates, since those are (well, should be) used to gracefully shutdown old instances when we update the worker type definition.

+ deferAuth: true,
+ scopes: [['aws-provisioner:manage-ami-set:<amiSetId>']],
+ input: undefined, // No input
+ output: undefined, // No output

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

You can just omit these

@djmitche

djmitche Jun 6, 2016

Contributor

You can just omit these

src/api-v1.js
+ let that = this;
+ let amiSet = req.params.amiSetId;
+
+ if (!req.satisfies({amiSet: amiSetId})) { return undefined; }

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

Why undefined?

@djmitche

djmitche Jun 6, 2016

Contributor

Why undefined?

This comment has been minimized.

@anarute

anarute Jun 6, 2016

Contributor

ops, sorry, I think I misunderstood this from worker-type.js

@anarute

anarute Jun 6, 2016

Contributor

ops, sorry, I think I misunderstood this from worker-type.js

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

An understandable confusion -- WorkerType.create actually overrides the Entity.create method to give it a different function signature. That's not necessary for AmiSet.

@djmitche

djmitche Jun 6, 2016

Contributor

An understandable confusion -- WorkerType.create actually overrides the Entity.create method to give it a different function signature. That's not necessary for AmiSet.

src/main.js
@@ -11,6 +11,8 @@ let base = require('taskcluster-base');
let workerType = require('./worker-type');
let secret = require('./secret');
+let workerState = require('./worker-state');

This comment has been minimized.

@jhford

jhford Jun 6, 2016

Collaborator

You should rebase your patch to the latest version of the upstream master branch. There was a change that I landed today that deletes the WorkerState entity, and it looks like you're adding it back here.

@jhford

jhford Jun 6, 2016

Collaborator

You should rebase your patch to the latest version of the upstream master branch. There was a change that I landed today that deletes the WorkerState entity, and it looks like you're adding it back here.

src/main.js
+ },
+ },
+
+ amiSet: {

This comment has been minimized.

@jhford

jhford Jun 6, 2016

Collaborator

We have a convention of anything that's a taskcluster-base.Entity using class casing, so please call this AmiSet here, and the other places that you are referring to the thing returned by AmiSet.setup instead of an instance of an AmiSet.

@jhford

jhford Jun 6, 2016

Collaborator

We have a convention of anything that's a taskcluster-base.Entity using class casing, so please call this AmiSet here, and the other places that you are referring to the thing returned by AmiSet.setup instead of an instance of an AmiSet.

src/main.js
+ setup: async ({cfg}) => {
+ let amiSet = amiSet.setup({
+ account: cfg.azure.account,
+ table: cfg.app.workerStateTableName,

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

I think you'll need a different table name for ami sets :)

@djmitche

djmitche Jun 6, 2016

Contributor

I think you'll need a different table name for ami sets :)

src/ami-set.js
+ * by its AWS region.
+ */
+
+let amiSet = base.Entity.configure({

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

This should be capitalized (AmiSet). The idea is that this is a class, so just like in Python we use an initial capital letter.

@djmitche

djmitche Jun 6, 2016

Contributor

This should be capitalized (AmiSet). The idea is that this is a class, so just like in Python we use an initial capital letter.

src/ami-set.js
+ amis: base.Entity.types.JSON,
+
+ },
+});

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

The content of this module looks OK, but it doesn't export anything. There's some information on exports here:

https://www.sitepoint.com/understanding-module-exports-exports-node-js/

In this case, we want require('./ami-set') to return the configured entity, just like for require('./worker-type').

@djmitche

djmitche Jun 6, 2016

Contributor

The content of this module looks OK, but it doesn't export anything. There's some information on exports here:

https://www.sitepoint.com/understanding-module-exports-exports-node-js/

In this case, we want require('./ami-set') to return the configured entity, just like for require('./worker-type').

This comment has been minimized.

@anarute

anarute Jun 6, 2016

Contributor

oh, right! so, I think I also need an AmiSet.create function, right?

@anarute

anarute Jun 6, 2016

Contributor

oh, right! so, I think I also need an AmiSet.create function, right?

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

You get that one for free -- base.Entity.configure(..) creates a class for you with a few methods (create, update, remove, etc.). Sadly there's basically no documentation for that (https://bugzilla.mozilla.org/show_bug.cgi?id=1278385).

@djmitche

djmitche Jun 6, 2016

Contributor

You get that one for free -- base.Entity.configure(..) creates a class for you with a few methods (create, update, remove, etc.). Sadly there's basically no documentation for that (https://bugzilla.mozilla.org/show_bug.cgi?id=1278385).

src/main.js
+ },
+ },
+
+ amiSet: {

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

This should be capitalized (AmiSet).

@djmitche

djmitche Jun 6, 2016

Contributor

This should be capitalized (AmiSet).

src/main.js
+ });
+ return WorkerState;
+ },
+ },

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

I think this may be a merge error? @jhford just removed the WorkerState entity class. Awesome that you've rebased your changes against the latest master, by the way!

@djmitche

djmitche Jun 6, 2016

Contributor

I think this may be a merge error? @jhford just removed the WorkerState entity class. Awesome that you've rebased your changes against the latest master, by the way!

This comment has been minimized.

@anarute

anarute Jun 6, 2016

Contributor

yes, it is a merge error! probably it was from the commit last week and I didn't see this part was removed. Now I see the importance of sending small patches as soon as I can :)

@anarute

anarute Jun 6, 2016

Contributor

yes, it is a merge error! probably it was from the commit last week and I didn't see this part was removed. Now I see the importance of sending small patches as soon as I can :)

src/api-v1.js
+ ].join('\n'),
+}, async function (req, res) {
+ let input = req.body;
+ let amiSet = req.params.amiSetId;

This comment has been minimized.

@jhford

jhford Jun 6, 2016

Collaborator

I probably would call this amiSetId since you'll be creating an Entity and probably want this name for that, instead of aSet

@jhford

jhford Jun 6, 2016

Collaborator

I probably would call this amiSetId since you'll be creating an Entity and probably want this name for that, instead of aSet

src/main.js
- setup: async ({cfg, WorkerType, Secret, ec2, stateContainer, validator, publisher, influx}) => {
+ requires: ['cfg', 'WorkerType', 'WorkerState', 'amiSet', 'Secret', 'ec2', 'validator', 'publisher', 'influx'],
+ setup: async ({cfg, WorkerType, WorkerState, amiSet, Secret, ec2, validator, publisher, influx}) => {
+

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

As above, I don't think you want WorkerState in here (@jhford please correct me if I'm wrong!)

@djmitche

djmitche Jun 6, 2016

Contributor

As above, I don't think you want WorkerState in here (@jhford please correct me if I'm wrong!)

This comment has been minimized.

@jhford

jhford Jun 6, 2016

Collaborator

Nope, she doesn't. I think that a merge went a little wrong, because I just today deleted the WorkerState on the master branch.

@jhford

jhford Jun 6, 2016

Collaborator

Nope, she doesn't. I think that a merge went a little wrong, because I just today deleted the WorkerState on the master branch.

src/main.js
let reportInstanceStarted = series.instanceStarted.reporter(influx);
let router = await v1.setup({
context: {
WorkerType: WorkerType,
+ amiSet: amiSet,

This comment has been minimized.

@djmitche

djmitche Jun 6, 2016

Contributor

This too should be capitalized (AmiSet)

@djmitche

djmitche Jun 6, 2016

Contributor

This too should be capitalized (AmiSet)

@jhford

This comment has been minimized.

Show comment
Hide comment
@jhford

jhford Jun 6, 2016

Collaborator

@anarute really good first pass! Most of the things that are needed are here. It seems like you and @djmitche are talking about json-schema for the AmiSets, which is a great idea. Otherwise, looking great!

Collaborator

jhford commented Jun 6, 2016

@anarute really good first pass! Most of the things that are needed are here. It seems like you and @djmitche are talking about json-schema for the AmiSets, which is a great idea. Otherwise, looking great!

@djmitche

This comment has been minimized.

Show comment
Hide comment
@djmitche

djmitche Jun 6, 2016

Contributor

OK that was a lot of comments, but this looks good!

If I remember correctly, you were able to run the tests for the provisioner. Do they still run after your changes?

You will also need some tests in test-src/manage_ami_set_test.js, similar to test-src/manageworkertype_test.js (but again, much shorter, since so far AMI sets are a lot less complicated than worker types). If it would help you out, I can write up a test for one of the API methods as an example.

Contributor

djmitche commented Jun 6, 2016

OK that was a lot of comments, but this looks good!

If I remember correctly, you were able to run the tests for the provisioner. Do they still run after your changes?

You will also need some tests in test-src/manage_ami_set_test.js, similar to test-src/manageworkertype_test.js (but again, much shorter, since so far AMI sets are a lot less complicated than worker types). If it would help you out, I can write up a test for one of the API methods as an example.

@anarute

This comment has been minimized.

Show comment
Hide comment
@anarute

anarute Jun 7, 2016

Contributor

Thank you @djmitche and @jhford for the comments and feedback! It was really helpful to understand things better. I think (if I have not added more things to fix) that I've covered most part of the issues, could you please review it again?

Contributor

anarute commented Jun 7, 2016

Thank you @djmitche and @jhford for the comments and feedback! It was really helpful to understand things better. I think (if I have not added more things to fix) that I've covered most part of the issues, could you please review it again?

src/api-v1.js
title: 'Create new AMI Set',
stability: base.API.stability.stable,
description: [
'Create an AMI Set. An AMI Set is a collection of AMIs with a single name.',
].join('\n'),
}, async function (req, res) {
let input = req.body;
- let amiSet = req.params.amiSetId;
+ let AmiSetId = req.params.id;

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

AmiSet is a 'class', but this isn't referring to the class. This is referring to a string and so should be amiSetId or just id... either one is fine.

@jhford

jhford Jun 7, 2016

Collaborator

AmiSet is a 'class', but this isn't referring to the class. This is referring to a string and so should be amiSetId or just id... either one is fine.

src/api-v1.js
return;
}
// Create amiSet
- let aSet;
+ let amiSetId;

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

this will refer to an AmiSet instance, so let's call it amiSet.

Basically, if a thing refers to a class (or in JS terms a constructor), it should be Caps

Here are some examples:

class ExampleClass { }

function ExampleConstructor () { }

let ExampleClass = require('./lib/example-class').ExampleClass;
let exampleInstance = new ExampleClass();

let exampleString = 'john';
@jhford

jhford Jun 7, 2016

Collaborator

this will refer to an AmiSet instance, so let's call it amiSet.

Basically, if a thing refers to a class (or in JS terms a constructor), it should be Caps

Here are some examples:

class ExampleClass { }

function ExampleConstructor () { }

let ExampleClass = require('./lib/example-class').ExampleClass;
let exampleInstance = new ExampleClass();

let exampleString = 'john';
src/main.js
account: cfg.azure.account,
- table: cfg.app.workerStateTableName,
+ table: cfg.app.amiSetTableName,

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

We'll need to add this variable to the config file config.yml

@jhford

jhford Jun 7, 2016

Collaborator

We'll need to add this variable to the config file config.yml

src/ami-set.js
+ id: base.Entity.types.String,
+ /* This is a JSON object which contains the AMIs of an AMI set keyed by
+ * their virtualization type and region. It is in the shape:
+ * {

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

This is not specific to your patch, but I'm sort of wondering if a structure like

{
   "us-west-1": {
      "hvm": "ami-111",
      "pv": "ami-222"
  },
   "us-east-1": {
      "hvm": "ami-111",
      "pv": "ami-222"
  }
}

or

[
   {
      "region": "us-west-1",
      "hvm": "ami-111",
      "pv": "ami-222"
   },
   {
      "region": "us-east-1",
      "hvm": "ami-111",
      "pv": "ami-222"
   }
]

The reason for flipping the hvm/pv around is that the structure goes from most general (region) to more specific (virtualization style) to most specific (ami id).

Whether we have a list of objects or a straigh mapping is something that you'll have to pick. A list of objects is really nice because you can do really neat things with Array.prototype.map and Array.prototype.filter. A mapping is really nice because it's easier to address the values and it is impossible to have duplicates. Error checking with the object will be easier, but the list of objects is more in line with what we do elsewhere in the provisioner and probably what I'd prefer to see.

@jhford

jhford Jun 7, 2016

Collaborator

This is not specific to your patch, but I'm sort of wondering if a structure like

{
   "us-west-1": {
      "hvm": "ami-111",
      "pv": "ami-222"
  },
   "us-east-1": {
      "hvm": "ami-111",
      "pv": "ami-222"
  }
}

or

[
   {
      "region": "us-west-1",
      "hvm": "ami-111",
      "pv": "ami-222"
   },
   {
      "region": "us-east-1",
      "hvm": "ami-111",
      "pv": "ami-222"
   }
]

The reason for flipping the hvm/pv around is that the structure goes from most general (region) to more specific (virtualization style) to most specific (ami id).

Whether we have a list of objects or a straigh mapping is something that you'll have to pick. A list of objects is really nice because you can do really neat things with Array.prototype.map and Array.prototype.filter. A mapping is really nice because it's easier to address the values and it is impossible to have duplicates. Error checking with the object will be easier, but the list of objects is more in line with what we do elsewhere in the provisioner and probably what I'd prefer to see.

This comment has been minimized.

@djmitche

djmitche Jun 7, 2016

Contributor

I tend to agree regarding the list of objects (the second option). That will also mean we don't need to change our schema when we add a new region. I also agree that we don't need to solve it in this patch.

@djmitche

djmitche Jun 7, 2016

Contributor

I tend to agree regarding the list of objects (the second option). That will also mean we don't need to change our schema when we add a new region. I also agree that we don't need to solve it in this patch.

src/main.js
@@ -124,13 +137,15 @@ let load = base.loader({
},
api: {
- requires: ['cfg', 'WorkerType', 'Secret', 'ec2', 'stateContainer', 'validator', 'publisher', 'influx'],
- setup: async ({cfg, WorkerType, Secret, ec2, stateContainer, validator, publisher, influx}) => {
+ requires: ['cfg', 'WorkerType', 'AmiSet', 'Secret', 'ec2', 'validator', 'publisher', 'influx'],

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

There's another merge issue here. stateContainer needs to be in both the requires list and the unpacked variable assignments in the setup function signature, but it's not. The usage of the stateContainer variable here isn't deleted, which is good.

@jhford

jhford Jun 7, 2016

Collaborator

There's another merge issue here. stateContainer needs to be in both the requires list and the unpacked variable assignments in the setup function signature, but it's not. The usage of the stateContainer variable here isn't deleted, which is good.

src/api-v1.js
@@ -580,6 +580,94 @@ api.declare({
+api.declare({
+ method: 'put',
+ route: '/ami-set/:amiSetId',

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

let's go with just :id here as well. When inside the managing functions for a thing itself, i find it easier to just refer to it as 'id' or something, and use more descriptive names for the ids of things that are referenced as a part of a thing.

@jhford

jhford Jun 7, 2016

Collaborator

let's go with just :id here as well. When inside the managing functions for a thing itself, i find it easier to just refer to it as 'id' or something, and use more descriptive names for the ids of things that are referenced as a part of a thing.

src/api-v1.js
+ input.lastModified = new Date();
+
+ // Authenticate request with parameterized scope
+ if (!req.satisfies({amiSet: AmiSetId})) {

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

the property that is checked against (here amiSet) needs to match the parameterized name in the scopes list (here amiSetId, but should be id).

After renaming the amiSetId variables and scopes to id, the object to use here would be {id: id}. In es6+, if your property name and value are derived from the save variable, e.g. let x = 1; {x: x} then you can just do `let x = 1; {x}.

@jhford

jhford Jun 7, 2016

Collaborator

the property that is checked against (here amiSet) needs to match the parameterized name in the scopes list (here amiSetId, but should be id).

After renaming the amiSetId variables and scopes to id, the object to use here would be {id: id}. In es6+, if your property name and value are derived from the save variable, e.g. let x = 1; {x: x} then you can just do `let x = 1; {x}.

src/api-v1.js
+ let amiSet = req.params.amiSetId;
+
+ try {
+ await this.amiSet.remove({amiSet: amiSetId}, true);

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

Remember that you're passing in the AmiSet entity class as a context variable to this function in src/main.js. This name here needs to be the name of the context object key, since what we do is take the 'context' object and assign all those values as properties on the this in this scope. In other words, you're looking for this.AmiSet.remove.

@jhford

jhford Jun 7, 2016

Collaborator

Remember that you're passing in the AmiSet entity class as a context variable to this function in src/main.js. This name here needs to be the name of the context object key, since what we do is take the 'context' object and assign all those values as properties on the this in this scope. In other words, you're looking for this.AmiSet.remove.

let reportInstanceStarted = series.instanceStarted.reporter(influx);
let router = await v1.setup({
context: {
WorkerType: WorkerType,
+ AmiSet: AmiSet,

This comment has been minimized.

@jhford

jhford Jun 7, 2016

Collaborator

Regarding my comment about this.AmiSet.remove, this is the place where you're creating the value this.AmiSet. This is magic done by the library, but fyi.

@jhford

jhford Jun 7, 2016

Collaborator

Regarding my comment about this.AmiSet.remove, this is the place where you're creating the value this.AmiSet. This is magic done by the library, but fyi.

@jhford

This comment has been minimized.

Show comment
Hide comment
@jhford

jhford Jun 10, 2016

Collaborator

I'm not sure what the scope of this PR is, but I noticed that there isn't a view or update endpoint for the AmiSets. If that's outside the scope of the PR as you've discussed with @djmitche , it's fine to hold off on that for the time being, but at some point we will need to add these :)

Collaborator

jhford commented Jun 10, 2016

I'm not sure what the scope of this PR is, but I noticed that there isn't a view or update endpoint for the AmiSets. If that's outside the scope of the PR as you've discussed with @djmitche , it's fine to hold off on that for the time being, but at some point we will need to add these :)

src/main.js
+ AmiSet: {
+ requires: ['cfg'],
+ setup: async ({cfg}) => {
+ let AmiSet = AmiSet.setup({

This comment has been minimized.

@jhford

jhford Jun 10, 2016

Collaborator

slight bug here. You are declaring a new variable, AmiSet, but assigning the first value to something that uses the variable declared here. This alone will cause

localhost:~/taskcluster/aws-provisioner $ node
> let x = x;
ReferenceError: x is not defined
    at repl:1:31
    at REPLServer.defaultEval (repl.js:252:27)
    at bound (domain.js:287:14)
    at REPLServer.runBound [as eval] (domain.js:300:12)
    at REPLServer.<anonymous> (repl.js:417:12)
    at emitOne (events.js:82:20)
    at REPLServer.emit (events.js:169:7)
    at REPLServer.Interface._onLine (readline.js:210:10)
    at REPLServer.Interface._line (readline.js:549:8)
    at REPLServer.Interface._ttyWrite (readline.js:826:14)

What you're looking for here is

      let AmiSet = amiSet.setup({ 

because you imported the AmiSet 'class' using the name amiSet above, where you require() the file.

@jhford

jhford Jun 10, 2016

Collaborator

slight bug here. You are declaring a new variable, AmiSet, but assigning the first value to something that uses the variable declared here. This alone will cause

localhost:~/taskcluster/aws-provisioner $ node
> let x = x;
ReferenceError: x is not defined
    at repl:1:31
    at REPLServer.defaultEval (repl.js:252:27)
    at bound (domain.js:287:14)
    at REPLServer.runBound [as eval] (domain.js:300:12)
    at REPLServer.<anonymous> (repl.js:417:12)
    at emitOne (events.js:82:20)
    at REPLServer.emit (events.js:169:7)
    at REPLServer.Interface._onLine (readline.js:210:10)
    at REPLServer.Interface._line (readline.js:549:8)
    at REPLServer.Interface._ttyWrite (readline.js:826:14)

What you're looking for here is

      let AmiSet = amiSet.setup({ 

because you imported the AmiSet 'class' using the name amiSet above, where you require() the file.

test-src/manageamiset_test.js
+
+ var id = slugid.nice();
+ var amiSetDefinition = makeAmiSet();
+ var amiSetChanged = _.clone(amiSetDefinition);

This comment has been minimized.

@jhford

jhford Jun 10, 2016

Collaborator

Without an update endpoint, this variable is not really useful. For the worker type tests, we have this changed variable where we basically create a second version of the tested definition, with one value changed. We then use the update method to verify that this expected value matches the one out of the api/table storage. For now, we should either delete this variable or we should have the update logic implemented.

@jhford

jhford Jun 10, 2016

Collaborator

Without an update endpoint, this variable is not really useful. For the worker type tests, we have this changed variable where we basically create a second version of the tested definition, with one value changed. We then use the update method to verify that this expected value matches the one out of the api/table storage. For now, we should either delete this variable or we should have the update logic implemented.

@djmitche

This comment has been minimized.

Show comment
Hide comment
@djmitche

djmitche Jun 12, 2016

Contributor

Yes, we should have "get" and "list" endpoints too. I feel silly for not noticing that! :)

Contributor

djmitche commented Jun 12, 2016

Yes, we should have "get" and "list" endpoints too. I feel silly for not noticing that! :)

@djmitche

This comment has been minimized.

Show comment
Hide comment
@djmitche

djmitche Jun 20, 2016

Contributor

From talking to @jhford and @anarute today, summarizing the remaining work:

  • get and list endpoints, plus tests
  • schema validation for the [{region, hvm, pv}, ..] format
Contributor

djmitche commented Jun 20, 2016

From talking to @jhford and @anarute today, summarizing the remaining work:

  • get and list endpoints, plus tests
  • schema validation for the [{region, hvm, pv}, ..] format
@jhford

This comment has been minimized.

Show comment
Hide comment
@jhford

jhford Jun 20, 2016

Collaborator

Oh, and also an update endpoint.

Collaborator

jhford commented Jun 20, 2016

Oh, and also an update endpoint.

+ debug(err.stack);
+ }
+ throw err;
+ }

This comment has been minimized.

@djmitche

djmitche Jun 20, 2016

Contributor

It looks like elements of list here would have a different format than that returned from getAmiSet, since listAmiSets is using item.ami.json(). The least surprising thing for a user of an API is if the two data structures have the same format. Using JSON schemas will help enforce this.

@djmitche

djmitche Jun 20, 2016

Contributor

It looks like elements of list here would have a different format than that returned from getAmiSet, since listAmiSets is using item.ami.json(). The least surprising thing for a user of an API is if the two data structures have the same format. Using JSON schemas will help enforce this.

This comment has been minimized.

@jhford

jhford Jun 21, 2016

Collaborator

I would almost say that we should just return a list of names, since that's what we do for the worker types.

@jhford

jhford Jun 21, 2016

Collaborator

I would almost say that we should just return a list of names, since that's what we do for the worker types.

This comment has been minimized.

@djmitche

djmitche Jun 21, 2016

Contributor

It's done either way in various places in TaskCluster. I think the key factors are how much work it is to fetch all of the data, and how long the response might get. In this case, scanning the table to get the names is no less work than scanning the table to get all of the AMI set content; and the resulting response is likely to be a fairly small fixed length (dozens of ami sets), so it's best to return the whole AMI set.

@djmitche

djmitche Jun 21, 2016

Contributor

It's done either way in various places in TaskCluster. I think the key factors are how much work it is to fetch all of the data, and how long the response might get. In this case, scanning the table to get the names is no less work than scanning the table to get all of the AMI set content; and the resulting response is likely to be a fairly small fixed length (dozens of ami sets), so it's best to return the whole AMI set.

This comment has been minimized.

@anarute

anarute Jun 23, 2016

Contributor

Should I return amiSet.amis.json() in the getAmiSet then?

@anarute

anarute Jun 23, 2016

Contributor

Should I return amiSet.amis.json() in the getAmiSet then?

This comment has been minimized.

@djmitche

djmitche Jun 23, 2016

Contributor

That would refer to a json() method on the amis property of the azure entity. Looking in ami-set.js, no such method is defined.

It probably makes sense to define an AmiSet.json() method which will return the JSON-able value for the whole AMI set -- including lastModified and id. Then you can call that method in both getAmiSet and listAmiSets.

@djmitche

djmitche Jun 23, 2016

Contributor

That would refer to a json() method on the amis property of the azure entity. Looking in ami-set.js, no such method is defined.

It probably makes sense to define an AmiSet.json() method which will return the JSON-able value for the whole AMI set -- including lastModified and id. Then you can call that method in both getAmiSet and listAmiSets.

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

This method is looking great now.

@djmitche

djmitche Jun 26, 2016

Contributor

This method is looking great now.

This comment has been minimized.

@jhford

jhford Jun 28, 2016

Collaborator

The provisioner mostly uses list of strings for list methods. I'd like to stick to that here unless there's a compelling reason not to. That would also make having a json-schema for this endpoint pretty quick to write up.

@jhford

jhford Jun 28, 2016

Collaborator

The provisioner mostly uses list of strings for list methods. I'd like to stick to that here unless there's a compelling reason not to. That would also make having a json-schema for this endpoint pretty quick to write up.

src/ami-set.js
+ */
+AmiSet.listAmiSets = async function () {
+
+ let amislist = [];

This comment has been minimized.

@jhford

jhford Jun 21, 2016

Collaborator

Normally, we'd camel case this into amisList, but given that a list is an aggregation(in this case, a list), I'd normally use a singular for the first part, so amiList, but since we're talking about ami sets and not amis, I'd call it amiSetList.

@jhford

jhford Jun 21, 2016

Collaborator

Normally, we'd camel case this into amisList, but given that a list is an aggregation(in this case, a list), I'd normally use a singular for the first part, so amiList, but since we're talking about ami sets and not amis, I'd call it amiSetList.

schemas/create-ami-set-request.yml
+ pv:
+ type: string
+ description: |
+ The AMI that uses PV virtualization type

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

you'll want

additionalProperties: false
required:
  - region
  - hvm
  - pv

here, too.

@djmitche

djmitche Jun 26, 2016

Contributor

you'll want

additionalProperties: false
required:
  - region
  - hvm
  - pv

here, too.

schemas/get-ami-set-response.yml
+ pv:
+ type: string
+ description: |
+ The AMI that uses PV virtualization type

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

Same (additionalProperties/required) here, too.

@djmitche

djmitche Jun 26, 2016

Contributor

Same (additionalProperties/required) here, too.

This comment has been minimized.

@jhford

jhford Jun 29, 2016

Collaborator

wait... why? pv is a type of string, it doesn't have properties... am I missing something?

@jhford

jhford Jun 29, 2016

Collaborator

wait... why? pv is a type of string, it doesn't have properties... am I missing something?

This comment has been minimized.

@djmitche

djmitche Jun 29, 2016

Contributor

Oh, indentation is a bit funny here, too -- it says amis has properties, rather than the items in amis. My comment was indicating that the region, hvm, and pv properties should be limited and all required.

@djmitche

djmitche Jun 29, 2016

Contributor

Oh, indentation is a bit funny here, too -- it says amis has properties, rather than the items in amis. My comment was indicating that the region, hvm, and pv properties should be limited and all required.

This comment has been minimized.

@jhford

jhford Jun 29, 2016

Collaborator

ahh right, so just intending it a bit. amis is an array, so it shouldn't have any properties. I'm surprised this schema works

@jhford

jhford Jun 29, 2016

Collaborator

ahh right, so just intending it a bit. amis is an array, so it shouldn't have any properties. I'm surprised this schema works

This comment has been minimized.

@jhford

jhford Jun 29, 2016

Collaborator

http://jsonschemalint.com/draft4/ is your friend. I patched it to support YAML serialised schemas and documents

@jhford

jhford Jun 29, 2016

Collaborator

http://jsonschemalint.com/draft4/ is your friend. I patched it to support YAML serialised schemas and documents

src/api-v1.js
+ input.lastModified = new Date();
+
+ // Authenticate request with parameterized scope
+ if (!req.satisfies({id: id})) {

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

(nit) {id}

@djmitche

djmitche Jun 26, 2016

Contributor

(nit) {id}

This comment has been minimized.

@anarute

anarute Jun 26, 2016

Contributor

I didn't understand, is {id} a short form for {id: id}?

@anarute

anarute Jun 26, 2016

Contributor

I didn't understand, is {id} a short form for {id: id}?

This comment has been minimized.

@djmitche

djmitche Jun 27, 2016

Contributor

Yep: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer

With ECMAScript 2015, there is a shorter notation available to achieve the same:

var a = "foo", 
    b = 42, 
    c = {};

// Shorthand property names (ES6)
var o = { a, b, c };
@djmitche

djmitche Jun 27, 2016

Contributor

Yep: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer

With ECMAScript 2015, there is a shorter notation available to achieve the same:

var a = "foo", 
    b = 42, 
    c = {};

// Shorthand property names (ES6)
var o = { a, b, c };
src/api-v1.js
+ for (let key of Object.keys(input)) {
+ amiS[key] = input[key];
+ amiS.lastModified = modDate;
+ }

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

There's really only one key of interest here (amis). The lastModified doesn't need to be copied from input (and input.lastModified doesn't need to be set, above, either). So while this works, it's a bit over-complicated.

@djmitche

djmitche Jun 26, 2016

Contributor

There's really only one key of interest here (amis). The lastModified doesn't need to be copied from input (and input.lastModified doesn't need to be set, above, either). So while this works, it's a bit over-complicated.

This comment has been minimized.

@anarute

anarute Jun 26, 2016

Contributor

should I just remove this logic for now?

@anarute

anarute Jun 26, 2016

Contributor

should I just remove this logic for now?

This comment has been minimized.

@djmitche

djmitche Jun 27, 2016

Contributor

Simplify, yes (basically just amiS.amis = input.amis)

@djmitche

djmitche Jun 27, 2016

Contributor

Simplify, yes (basically just amiS.amis = input.amis)

src/api-v1.js
+ output: 'get-ami-set-response.json#',
+ deferAuth: true,
+ scopes: [
+ ['aws-provisioner:view-ami-set:<amiSetId>'],

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

Here, getAmiSet requires aws-provisioner:view-ami-set:<id>, but listAmiSets requires no scopes. It's tricky to enforce scopes in a list method, and I don't see a great reason to hide AMI sets from public view, so I think we should just allow anyone to call getAmiSet, too (meaning, remove the scopes: here and the req.satisifies below).

@djmitche

djmitche Jun 26, 2016

Contributor

Here, getAmiSet requires aws-provisioner:view-ami-set:<id>, but listAmiSets requires no scopes. It's tricky to enforce scopes in a list method, and I don't see a great reason to hide AMI sets from public view, so I think we should just allow anyone to call getAmiSet, too (meaning, remove the scopes: here and the req.satisifies below).

test-src/manageamiset_test.js
+
+ // var id = slugid.nice();
+ var amiSetDefinition = makeAmiSet();
+ var amiSetChanged = _.clone(amiSetDefinition);

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

It doesn't look like this AMI set is actually different from the original. Maybe add a new AMI to it, just to be more realistic?

@djmitche

djmitche Jun 26, 2016

Contributor

It doesn't look like this AMI set is actually different from the original. Maybe add a new AMI to it, just to be more realistic?

+ } catch (err) {
+ assume(err.statusCode).equals(404);
+ }
+ });

This comment has been minimized.

@djmitche

djmitche Jun 26, 2016

Contributor

I don't see a test for listAmiSets..

@djmitche

djmitche Jun 26, 2016

Contributor

I don't see a test for listAmiSets..

@djmitche

This comment has been minimized.

Show comment
Hide comment
@djmitche

djmitche Jun 26, 2016

Contributor

John's review is the one that counts, but as far as I can see this is looking good with just a few more minor things to fix:

  • tests for listAmiSets
  • actually change the AMI set in tests
  • schema fixes
  • remove scopes for getAmiSets

Once you get going writing tools and backend support for this, you'll probably find some other things to change, but that's easy to do with another PR.

Contributor

djmitche commented Jun 26, 2016

John's review is the one that counts, but as far as I can see this is looking good with just a few more minor things to fix:

  • tests for listAmiSets
  • actually change the AMI set in tests
  • schema fixes
  • remove scopes for getAmiSets

Once you get going writing tools and backend support for this, you'll probably find some other things to change, but that's easy to do with another PR.

@anarute

This comment has been minimized.

Show comment
Hide comment
@anarute

anarute Jun 26, 2016

Contributor

Cool! I'm working on this changes and I'll rebase these commits to squash the "fix" ones when I finish them

Contributor

anarute commented Jun 26, 2016

Cool! I'm working on this changes and I'll rebase these commits to squash the "fix" ones when I finish them

test-src/manageamiset_test.js
+ it('should return a list of AMI sets', async () => {
+ await client.listAmiSets();
+ });
+

This comment has been minimized.

@djmitche

djmitche Jun 27, 2016

Contributor

does it actually return a list of AMIs? :)

@djmitche

djmitche Jun 27, 2016

Contributor

does it actually return a list of AMIs? :)

This comment has been minimized.

@djmitche

djmitche Jun 27, 2016

Contributor

Per out discussion, this should use assume(amiSets).to.deeply.equal(..).

@djmitche

djmitche Jun 27, 2016

Contributor

Per out discussion, this should use assume(amiSets).to.deeply.equal(..).

src/ami-set.js
+ try {
+ await base.Entity.scan.call(this, {}, {
+ handler: function(item) {
+ amiSetList.push(item.json());

This comment has been minimized.

@jhford

jhford Jun 28, 2016

Collaborator

If this is a listing method, we should be pushing the item.id property instead of the complete item in a json-serializable format. If the goal is a load-all method, let's call it loadAll() to match the worker type here

@jhford

jhford Jun 28, 2016

Collaborator

If this is a listing method, we should be pushing the item.id property instead of the complete item in a json-serializable format. If the goal is a load-all method, let's call it loadAll() to match the worker type here

This comment has been minimized.

@djmitche

djmitche Jun 28, 2016

Contributor

I believe we already had some discussion of this in the PR? Many of the list methods in TC services return the entire object, e.g., auth.listClients. The reason not to do so is if it requires additional queries or a lot more bandwidth. Neither is the case here. We have the additional information about each AMI set returned from Azure, so why not provide it to the user?

@djmitche

djmitche Jun 28, 2016

Contributor

I believe we already had some discussion of this in the PR? Many of the list methods in TC services return the entire object, e.g., auth.listClients. The reason not to do so is if it requires additional queries or a lot more bandwidth. Neither is the case here. We have the additional information about each AMI set returned from Azure, so why not provide it to the user?

src/api-v1.js
+ 'id',
+ 'amis',
+ ].every((key) => {
+ return _.isEqual(amiSet[key], input[key]);

This comment has been minimized.

@jhford

jhford Jun 28, 2016

Collaborator

I can't remember off hand, but we should ensure that _.isEqual is a deep equality operator, since an amiSet is an object which collects other objects.

@jhford

jhford Jun 28, 2016

Collaborator

I can't remember off hand, but we should ensure that _.isEqual is a deep equality operator, since an amiSet is an object which collects other objects.

This comment has been minimized.

src/api-v1.js
+api.declare({
+ method: 'get',
+ route: '/ami-set/:id',
+ name: 'getAmiSet',

This comment has been minimized.

@jhford

jhford Jun 28, 2016

Collaborator

The WorkerType entity uses name: 'workerType'. Can we follow that pattern here?

@jhford

jhford Jun 28, 2016

Collaborator

The WorkerType entity uses name: 'workerType'. Can we follow that pattern here?

src/api-v1.js
+
+ let amiSet = await this.AmiSet.load({id: id});
+
+ await amiSet.modify(function(amiS) {

This comment has been minimized.

@jhford

jhford Jun 28, 2016

Collaborator

the name amiS sort of implies that it's a plural of amis but with confusing captalization. Instead, since we're passing in an instance of an AmiSet, let's use the amiSet variable name.

@jhford

jhford Jun 28, 2016

Collaborator

the name amiS sort of implies that it's a plural of amis but with confusing captalization. Instead, since we're passing in an instance of an AmiSet, let's use the amiSet variable name.

+ });
+
+ beforeEach(async () => {
+ await main('tableCleaner', {process: 'tableCleaner', profile: 'test'});

This comment has been minimized.

@jhford

jhford Jun 28, 2016

Collaborator

We should add the AmiSet entity to the tableCleaner component in src/main.js so that this table is actually cleaned out.

@jhford

jhford Jun 28, 2016

Collaborator

We should add the AmiSet entity to the tableCleaner component in src/main.js so that this table is actually cleaned out.

test-src/manageamiset_test.js
+ amiSet = await client.getAmiSet(id);
+ lastModified = amiSet.lastModified;
+ assume(await amiSet).to.deeply.equal({
+ amis: [

This comment has been minimized.

@jhford

jhford Jun 28, 2016

Collaborator

we use mock objects in a single file elsewhere to make it easier to update the format of the test objects without having to completely rewrite the test. In this case, the objects are simple enough that I don't know if it's worth doing here, but please take a look at the test-src/mock*.js files just to see how they're structured. Basically, we have a default that is returned, but you can overwrite the values you want overwritten. Not critical here, but worth looking at.

@jhford

jhford Jun 28, 2016

Collaborator

we use mock objects in a single file elsewhere to make it easier to update the format of the test objects without having to completely rewrite the test. In this case, the objects are simple enough that I don't know if it's worth doing here, but please take a look at the test-src/mock*.js files just to see how they're structured. Basically, we have a default that is returned, but you can overwrite the values you want overwritten. Not critical here, but worth looking at.

@anarute

This comment has been minimized.

Show comment
Hide comment
@anarute

anarute Jun 28, 2016

Contributor

@jhford I fixed the things you've mentioned:

Contributor

anarute commented Jun 28, 2016

@jhford I fixed the things you've mentioned:

@djmitche

This comment has been minimized.

Show comment
Hide comment
@djmitche

djmitche Jun 28, 2016

Contributor

If we need loadAll(), it's easy enough to add in a new PR :)

Contributor

djmitche commented Jun 28, 2016

If we need loadAll(), it's easy enough to add in a new PR :)

@jhford

This comment has been minimized.

Show comment
Hide comment
@jhford

jhford Jun 29, 2016

Collaborator

So there's a problem here. The create ami set endpoint has a json-schema that forces there to be an ID parameter in the request body. The problem is that we already have that specified in the parameters of the api method. The create schema should only have the amis object.

Collaborator

jhford commented Jun 29, 2016

So there's a problem here. The create ami set endpoint has a json-schema that forces there to be an ID parameter in the request body. The problem is that we already have that specified in the parameters of the api method. The create schema should only have the amis object.

@jhford

This comment has been minimized.

Show comment
Hide comment
@jhford

jhford Jun 29, 2016

Collaborator

Ok. So I think the only issue left here is removing the createAmiSet json-schema allowing an ID value. The only place we should be supplying the id value is the parameter that we bake into the /ami-set/:id route.

I think with that change, i'm 99% sure that we're ready to merge!

Collaborator

jhford commented Jun 29, 2016

Ok. So I think the only issue left here is removing the createAmiSet json-schema allowing an ID value. The only place we should be supplying the id value is the parameter that we bake into the /ami-set/:id route.

I think with that change, i'm 99% sure that we're ready to merge!

@anarute

This comment has been minimized.

Show comment
Hide comment
@anarute

anarute Jun 30, 2016

Contributor

@jhford I removed the id property from createAmiSet schema, could you review the code again, please?

Contributor

anarute commented Jun 30, 2016

@jhford I removed the id property from createAmiSet schema, could you review the code again, please?

schemas/create-ami-set-request.yml
+ - region
+ - hvm
+ - pv
+ lastModified:

This comment has been minimized.

@jhford

jhford Jul 1, 2016

Collaborator

Sorry that I didn't see this before, but we also should not have last modified as an allowed field in the request. we will generate this value inside the api handler, so forcing a value here that we won't end up using isn't ideal.

@jhford

jhford Jul 1, 2016

Collaborator

Sorry that I didn't see this before, but we also should not have last modified as an allowed field in the request. we will generate this value inside the api handler, so forcing a value here that we won't end up using isn't ideal.

src/api-v1.js
+ let input = req.body;
+ let id = req.params.id;
+
+ input.lastModified = new Date();

This comment has been minimized.

@jhford

jhford Jul 1, 2016

Collaborator

We shouldn't really be setting properties on the request body object. Also, this value isn't actually used in the AmiSet.create method call below

@jhford

jhford Jul 1, 2016

Collaborator

We shouldn't really be setting properties on the request body object. Also, this value isn't actually used in the AmiSet.create method call below

+ amiSet = await this.AmiSet.create({
+ id: id,
+ amis: input.amis,
+ lastModified: new Date(),

This comment has been minimized.

@jhford

jhford Jul 1, 2016

Collaborator

here's where we're actually using this value

@jhford

jhford Jul 1, 2016

Collaborator

here's where we're actually using this value

This comment has been minimized.

@anarute

anarute Jul 1, 2016

Contributor

you're right, I passed that out among all the changes! thanks for noticing

@anarute

anarute Jul 1, 2016

Contributor

you're right, I passed that out among all the changes! thanks for noticing

@@ -0,0 +1,107 @@
+var slugid = require('slugid');

This comment has been minimized.

@jhford

jhford Jul 1, 2016

Collaborator

random nit, this file uses let and var. We should probably chose one and stick to it.

The difference in case you're unfamiliar is that var is in scope through the entire function. If you use var in an if, the variable is in scope through the entire function. Let is block scope, if you use let in an if, it's only in scope inside that if and those blocks (e.g. code inside of { }) that are nested.

Let is better than var! Also, if you're really curious do a search about 'variable hoisting' for more info on how var works.

@jhford

jhford Jul 1, 2016

Collaborator

random nit, this file uses let and var. We should probably chose one and stick to it.

The difference in case you're unfamiliar is that var is in scope through the entire function. If you use var in an if, the variable is in scope through the entire function. Let is block scope, if you use let in an if, it's only in scope inside that if and those blocks (e.g. code inside of { }) that are nested.

Let is better than var! Also, if you're really curious do a search about 'variable hoisting' for more info on how var works.

This comment has been minimized.

@anarute

anarute Jul 1, 2016

Contributor

thanks for the explanation!

@anarute

anarute Jul 1, 2016

Contributor

thanks for the explanation!

test-src/manageamiset_test.js
+ debug('### Load amiSet (again)');
+ amiSet = await client.amiSet(id);
+ lastModified = amiSet.lastModified;
+ assume(await amiSet).to.deeply.equal({

This comment has been minimized.

@jhford

jhford Jul 1, 2016

Collaborator

we don't need to await amiSet because we already await'd it above.

@jhford

jhford Jul 1, 2016

Collaborator

we don't need to await amiSet because we already await'd it above.

This comment has been minimized.

@jhford

jhford Jul 1, 2016

Collaborator

Also, i think this is way outside the scope of this project, but one thing that I like to do is compare the amiSetChanged object itself. Doing that does require that you handle the lastModified stuff, as you do correctly below. Another option is to use Sinon to mock out the system clock. That's really out of scope and what you have here is fine. I might consider using _.default to grab the amis object from amiSetChanged

@jhford

jhford Jul 1, 2016

Collaborator

Also, i think this is way outside the scope of this project, but one thing that I like to do is compare the amiSetChanged object itself. Doing that does require that you handle the lastModified stuff, as you do correctly below. Another option is to use Sinon to mock out the system clock. That's really out of scope and what you have here is fine. I might consider using _.default to grab the amis object from amiSetChanged

@jhford jhford merged commit 72a05c9 into taskcluster:master Jul 4, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment