Skip to content

Commit

Permalink
fresh => writableFilesystem (#234)
Browse files Browse the repository at this point in the history
* fresh => writableFilesystem

* Remove last reference to fresh
  • Loading branch information
Jake Pruitt committed Jun 27, 2018
1 parent f6bdbdd commit f273e76
Show file tree
Hide file tree
Showing 7 changed files with 12 additions and 12 deletions.
2 changes: 1 addition & 1 deletion bin/watchbot.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ const main = async () => {

const options = {
queueUrl: process.env.QueueUrl,
fresh: process.env.fresh === 'true' ? true : false,
writableFilesystem: process.env.writableFilesystem === 'true' ? true : false,
workerOptions: { command, volumes }
};

Expand Down
6 changes: 3 additions & 3 deletions docs/building-a-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ When creating your watchbot stacks with the `watchbot.template()` method, you no
**family** | The name of the task definition family that watchbot will create revisions of. | String/Ref | Yes | -
**command** | The shell command to be run by the subprocess worker. The working directory for the subprocess is determined in your Dockerfile by the `WORKDIR` missive. | String | Yes | -
**workers** | The maximum number of workers to run for your service. Must be a number, not a reference to a number, since one tenth of this number will be used as the scaling adjustment for the scaling policy. | Number | Yes | -
**fresh** | Whether you want a fresh container for every job. See below for more details. | Boolean | No | false
**writableFilesystem** | Whether you want a fresh container for every job with a writable filesystem. See below for more details. | Boolean | No | false
**mounts** | If your worker containers need to write files or folders inside its file system, specify those locations with this parameter. A single ephemeral mount point can be specified as `{container location}`, e.g. /mnt/tmp. Separate multiple mount strings with commas if you need to mount more than one location. You can also specify mounts as an arrays of paths. Every mounted volume will be cleaned after each job. By default, the `/tmp` directory is added as an ephemeral mount. | String/Object | No | `/tmp`
**env** | Key-value pairs that will be provided to the worker containers as environment variables. Keys must be strings, and values can either be strings or references to other CloudFormation resources via `{"Ref": "..."}`. | Object | No | `{}`
**prefix** | a prefix that will be applied to the logical names of all the resources Watchbot creates. If you're building a template that includes more than one Watchbot system, you'll need to specify this in order to differentiate the resources. | String/Ref | No | none
Expand All @@ -95,9 +95,9 @@ By default, containers are re-used from one job to the next, and file system is

Since containers are only started once during scale up and then left on for long durations, users can expect to see very few failed task placements. Combined with the low overhead of not needing to start containers for every job, watchbot is ideal for workloads that are potentially very short-lived and require high throughput. During initial benchmarks, watchbot was able to achieve a throughput of 50 tasks per second when run at 500 workers for jobs that ran 10 seconds each. There were no signs showing that it would slow down, and seemed to be able to handle as much throughput as you were willing to add workers.

**Fresh mode**
**writableFilesystem mode**

In fresh mode, containers are stopped after every job. This refreshing of containers allows users to confidently expect their work to run in a fresh container every time, and allow them to write to anywhere on the filesystem. Fresh mode throughput values have not been confirmed yet, but it can be guaranteed to be slower than the default mode, due to the overhead of starting a new container after every job.
In writableFilesystem mode, the whole file system is writable and containers are stopped after every job. This refreshing of containers allows users to confidently expect their work to run in a fresh container every time, and allow them to write to anywhere on the filesystem. Fresh mode throughput values have not been confirmed yet, but it can be guaranteed to be slower than the default mode, due to the overhead of starting a new container after every job.

Fresh mode has no restrictions to the file system: workers can write anywhere and read from anywhere, their files being instantly deleted after the job finishes and the container dies.

Expand Down
4 changes: 2 additions & 2 deletions lib/template.js
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ module.exports = (options = {}) => {
minSize: 0,
mounts: '',
privileged: false,
fresh: false,
writableFilesystem: false,
family: options.service,
errorThreshold: 10,
alarmThreshold: 40,
Expand All @@ -139,7 +139,7 @@ module.exports = (options = {}) => {
[
{ Name: 'WorkTopic', Value: cf.ref(prefixed('Topic')) },
{ Name: 'QueueUrl', Value: cf.ref(prefixed('Queue')) },
{ Name: 'fresh', Value: options.fresh },
{ Name: 'writableFilesystem', Value: options.writableFilesystem },
{ Name: 'Volumes', Value: mountPoints.map((m) => m.ContainerPath).join(',') }
]
);
Expand Down
4 changes: 2 additions & 2 deletions lib/watcher.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ class Watcher {
this.workerOptions = options.workerOptions;
this.queueUrl = options.queueUrl;
this.messages = Messages.create({ queueUrl: options.queueUrl });
this.freshMode = options.fresh;
this.writableFilesystem = options.writableFilesystem;
}

listen() {
Expand All @@ -32,7 +32,7 @@ class Watcher {
Worker.create(message, this.workerOptions).waitFor()
);
await Promise.all(workers);
return this.freshMode ? resolve() : setImmediate(loop);
return this.writableFilesystem ? resolve() : setImmediate(loop);
};
setImmediate(loop);
});
Expand Down
4 changes: 2 additions & 2 deletions test/__snapshots__/template.jest.js.snap
Original file line number Diff line number Diff line change
Expand Up @@ -623,7 +623,7 @@ Object {
},
},
Object {
"Name": "fresh",
"Name": "writableFilesystem",
"Value": false,
},
Object {
Expand Down Expand Up @@ -1395,7 +1395,7 @@ Object {
},
},
Object {
"Name": "fresh",
"Name": "writableFilesystem",
"Value": false,
},
Object {
Expand Down
2 changes: 1 addition & 1 deletion test/bin.watchbot.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ test('[bin.watchbot] success', async (assert) => {
assert.ok(
Watcher.create.calledWith({
queueUrl: 'https://faker',
fresh: false,
writableFilesystem: false,
workerOptions: {
command: 'echo hello world',
volumes: ['/tmp', '/mnt']
Expand Down
2 changes: 1 addition & 1 deletion test/watcher.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ test('[watcher] listens exactly once', async (assert) => {

const watcher = new Watcher({
queueUrl: 'https://faker',
fresh: true,
writableFilesystem: true,
workerOptions: {
command: 'echo hello world',
volumes: ['/tmp']
Expand Down

0 comments on commit f273e76

Please sign in to comment.