Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KETTLE-66: Provide a standard Kettle middleware wrapper for Multer #41

Merged
merged 74 commits into from Aug 16, 2018
Merged
Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
42a696f
KETTLE-66: add multer to package.json.
waharnum Dec 14, 2017
cb40bda
KETTLE-66: beginnings of multer testing using form-data.
waharnum Dec 15, 2017
3dca817
KETTLE-66: some basic working tests + implementation.
waharnum Dec 15, 2017
545d427
KETTLE-66: test for trying to upload files beyond an array maxlimit.
waharnum Dec 15, 2017
e8bcb26
KETTLE-66: add basic test coverage.
waharnum Dec 15, 2017
c778fd6
KETTLE-66: test for multer body.
waharnum Dec 15, 2017
416aaab
KETTLE-66: add multer tests to allTests.
waharnum Dec 15, 2017
faa5aa1
KETTLE-66: add test and implementation of file filter.
waharnum Dec 18, 2017
03ffded
KETTLE-66: update license files.
waharnum Dec 18, 2017
68096a0
KETTLE-66: invoker-based storage function.
waharnum Dec 21, 2017
3e91d7f
KETTLE-66: refactoring for invokers.
waharnum Dec 22, 2017
6d4c3b2
KETTLE-66: more refactoring to invokers.
waharnum Dec 22, 2017
1125130
KETTLE-66: rename test function.
waharnum Dec 22, 2017
dd7a9bb
KETTLE-66: remove unneeded multerConfig block.
waharnum Dec 22, 2017
bc3f101
KETTLE-66: abstract file filter function for mimetypes.
waharnum Dec 22, 2017
51aa3d4
KETTLE-66: rename invokers / functions.
waharnum Dec 22, 2017
74df39b
KETTLE-66: add options for using diskStorage, simplify invokers.
waharnum Jan 8, 2018
07d83d8
KETTLE-66: test upload directory.
waharnum Jan 8, 2018
daaf3e7
KETTLE-66: start adding tests for diskStorage.
waharnum Jan 9, 2018
bec4973
KETTLE-66: make a generic function that the multer tests can use as w…
waharnum Jan 9, 2018
84ee2b7
KETTLE-66: use ensureDirectoryEmpty for MulterTests setup.
waharnum Jan 10, 2018
9114cc5
KETTLE-66: resolve test file paths with fluid.module.resolvePath.
waharnum Jan 10, 2018
b6f56d8
KETTLE-66: test for presence of uploaded file.
waharnum Jan 10, 2018
b4f4d3d
KETTLE-66: in-code comments.
waharnum Jan 12, 2018
e9020fd
KETTLE-66: adding comments.
waharnum Jan 18, 2018
6d68d27
KETTLE-66: add limits options explicitly.
waharnum Jan 18, 2018
f24e11c
KETTLE-66: update copyrights, TODO on ensureWriteableEmpty replacement.
waharnum Jan 18, 2018
7c25bd9
KETTLE-66: linting.
waharnum Jan 18, 2018
e62e5d5
KETTLE-66: update field size to multer default of 1MB.
waharnum Mar 8, 2018
d8da195
Merge remote-tracking branch 'fluid-project/master' into KETTLE-66
waharnum Jun 5, 2018
7e111df
KETTLE-66: unwrap fileFilter per PR feedback.
waharnum Jun 19, 2018
6d6720f
KETTLE-66: refactor diskStorage destination/filename invokers per fee…
waharnum Jun 19, 2018
ba3f0ef
KETTLE-66: shift construction of diskStorage and memoryStorage onto m…
waharnum Jun 19, 2018
11eb539
Merge branch 'KETTLE-66' of https://github.com/waharnum/kettle into K…
waharnum Jun 20, 2018
c2c20e7
KETTLE-66: add appropriate cleanup listeners for tests involving file…
waharnum Jun 20, 2018
2a66e0e
KETTLE-66: remove old TODO statement.
waharnum Jun 20, 2018
7432a25
KETTLE-66: update explanatory TODO.
waharnum Jun 20, 2018
ad0d5af
KETTLE-66: store a diskStorage configuration object on the config mat…
waharnum Jun 20, 2018
1cded9e
KETTLE-66: linting.
waharnum Jun 20, 2018
b8d2270
KETTLE-66: move test file paths into a holder component on the test s…
waharnum Jun 20, 2018
9270c1c
KETTLE-66: kettle.test.request.http refactoring per PR feedback - ref…
waharnum Jun 20, 2018
7ee255d
KETTLE-66: code comments about new test.request.http.send workflow.
waharnum Jun 20, 2018
75d4d34
KETTLE-66: linting.
waharnum Jun 20, 2018
b04bfc5
KETTLE-66: refactoring kettle.test.request.formData - extracting to s…
waharnum Jun 21, 2018
e0f3aff
KETTLE-66: remove old formData.send method.
waharnum Jun 21, 2018
8325d02
KETTLE-66: simplify file appending loop.
waharnum Jun 21, 2018
28a7b6f
KETTLE-66: refactoring to increase configurability of multer function…
waharnum Jun 21, 2018
dd7cbc0
KETTLE-66: rename getMiddlewareForFileStrategy -> getMulterMethod.
waharnum Jun 21, 2018
6d86f98
KETTLE-66: remove getMulterMethod, merge its functions into main crea…
waharnum Jun 21, 2018
e690924
KETTLE-66: add buildkite configuration.
waharnum Jun 21, 2018
c445574
KETTLE-66: add buildkite config to correct directory structure. :/
waharnum Jun 21, 2018
d1637d2
KETTLE-66: start writing documentation.
waharnum Jun 22, 2018
73228e3
KETTLE-66: continue documentation.
waharnum Jun 22, 2018
bb2f3cb
KETTLE-66: adding example for multiPartForm
waharnum Jun 25, 2018
41c6120
KETTLE-66: add to documentation.
waharnum Jun 25, 2018
b17219e
KETTLE-66: default to POST method for kettle.test.request.formData.
waharnum Jun 27, 2018
6c589d7
KETTLE-66: merge upstream master, resolve some conflicts in package.json
waharnum Jul 12, 2018
b1da90e
KETTLE-66: merge conflicts.
waharnum Jul 12, 2018
11f9ffa
KETTLE-66: merge upstream, remove package-lock.json.
waharnum Aug 1, 2018
4173f6f
Merge branch 'KETTLE-66' of https://github.com/waharnum/kettle into K…
waharnum Aug 1, 2018
cd1a811
KETTLE-66: middle of refactoring, disk storage not working yet.
waharnum Aug 1, 2018
de616cf
KETTLE-66: initial refactor done, tests passing.
waharnum Aug 1, 2018
84f7322
KETTLE-66: allow fluid.stringTemplate syntax to be used in destinatio…
waharnum Aug 1, 2018
39d7db6
KETTLE-66: update example in light of refactoring.
waharnum Aug 1, 2018
1f984db
KETTLE-66: extract fileFilter functionality to a component grade.
waharnum Aug 1, 2018
9781571
KETTLE-66: update documentation and add comments.
waharnum Aug 1, 2018
ac82c0a
KETTLE-66: update buildkite pipeline config.
waharnum Aug 1, 2018
4d100f4
KETTLE-66: use argument-shifting invokers, avoid callback-style code.
waharnum Aug 1, 2018
7deec66
KETTLE-66: code comments.
waharnum Aug 1, 2018
ba3b996
KETTLE-66: update code comment.
waharnum Aug 1, 2018
c5abad0
KETTLE-66: don't share names between configuration options and invoke…
waharnum Aug 15, 2018
74a98ac
KETTLE-66: break all comments at 80 characters.
waharnum Aug 15, 2018
6fcb747
KETTLE-66: use *Resolver for invokers that result in Multer-style cal…
waharnum Aug 15, 2018
7606c18
KETTLE-66: use *Resolver convention for fileFilter invoker.
waharnum Aug 15, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Expand Up @@ -5,3 +5,5 @@ yarn.lock

/coverage
/node_modules
/examples/multipartForm/uploads/*
!/examples/multipartForm/uploads/.gitkeep
8 changes: 8 additions & 0 deletions docs/KettleTestingFramework.md
Expand Up @@ -302,6 +302,14 @@ Consult the framework tests at [tests/shared/SessionTestDefs.js](../tests/shared
These tests are also a good example of configuring custom [middleware](#working-with-middleware) into the middle of a request's middleware chain. These tests include a middleware grade named `kettle.tests.middleware.validateSession` which
will reject requests without a particular piece of populated session data, before processing reaches the request's `requestHandler`.

### Sending multipart-form testing fixtures via POST with `kettle.test.request.form`

The `kettle.test.request.form` grade derives from `kettle.test.request.http`, but accepts additional configuration for sending `multipart-form-data` requests that can include a mix of fields and files.

### Configuration options available on `kettle.test.request.form`

<a href="#configuration-options-available-on-kettletestrequesthttp">In addition to all options from `kettle.test.request.http`</a>, the grade accepts the following options:

### Issuing WebSockets testing fixtures with `kettle.test.request.ws`

A sibling grade of `kettle.test.request.http` is `kettle.test.request.ws` which will allow the testing of WebSockets endpoints in an analogous way. You should browse the `ws` project's documentation for
Expand Down
86 changes: 85 additions & 1 deletion docs/Middleware.md
Expand Up @@ -161,6 +161,13 @@ to implement your own.
<td><code>secret</code> and <code>middlewareOptions</code>, forwarded to the two arguments of <a href="https://github.com/expressjs/cookie-parser#cookieparsersecret-options"><code>cookieParser(secret, options)</code></a></td>
<td>none</td>
</tr>
<tr>
<td><code>kettle.middleware.multer</code></td>
<td><a href="https://github.com/expressjs/multer">expressjs/multer</a></td>
<td>Handles <code>multipart/form-data</code>, primarily for file uploading.</td>
<td><code>middlewareOptions</code>, forwarded to <code>multer(options)</code>, and <code>formFieldOptions</code>, used to configure the field parameters for uploaded files as described in <a href="https://github.com/expressjs/multer#usage">multer's documentation</a>. <strong>Note</strong>: some <code>multer</code> options require functions as their values, and are implemented in Kettle using <code>invokers</code>; see the documentation below on using <code>kettle.middleware.multer</code> for more details.</td>
<td>none – user must configure on each use</td>
</tr>
<tr>
<td><code>kettle.middleware.session</code></td>
<td><a href="https://github.com/expressjs/session">expressjs/session</a></td>
Expand Down Expand Up @@ -265,5 +272,82 @@ fluid.defaults("examples.static.handler", {
}
}
});
```

#### Using the multer middleware

This shows a basic single-file upload; for more examples of possible usage, refer to the `kettle.tests.multer.config.json5` configuration file in `tests/configs` and the Multer documentation.

Code for this example can be found in `/examples/multipartForm`.

```javascript
fluid.defaults("examples.uploadConfig", {
"gradeNames": ["fluid.component"],
"components": {
"server": {
"type": "kettle.server",
"options": {
"port": 8081,
"components": {
"imageUpload": {
"type": "kettle.middleware.multer",
"options": {
"formFieldOptions": {
"method": "single",
"fieldName": "image"
},
"members": {
"storage": "{that}.diskStorage"
},
"invokers": {
"diskStorageDestination": {
"funcName": "examples.uploadConfig.diskStorageDestination"
}
}
}
},
"app": {
"type": "kettle.app",
"options": {
"requestHandlers": {
"imageUploadHandler": {
"type": "examples.uploadConfig.handler",
"route": "/upload",
"method": "post"
}
}
}
}
}
}
}
}
});
```

And our corresponding handlers and `diskStorageDestination` invoker (to set our own destination path):

```javascript
examples.uploadConfig.diskStorageDestination = function (req, file, cb) {
cb(null, "./examples/multipartForm/uploads");
};

fluid.defaults("examples.uploadConfig.handler", {
gradeNames: "kettle.request.http",
requestMiddleware: {
imageUpload: {
middleware: "{server}.imageUpload"
}
},
invokers: {
handleRequest: "examples.uploadConfig.handleRequest"
}
});

```
examples.uploadConfig.handleRequest = function (request) {
var uploadedFileDetails = request.req.file;
request.events.onSuccess.fire({
message: fluid.stringTemplate("POST request received on path /upload; file %originalName uploaded to %uploadedPath", {originalName: uploadedFileDetails.originalname, uploadedPath: uploadedFileDetails.path})
});
};
```
44 changes: 44 additions & 0 deletions examples/multiPartForm/multiPartForm-client.js
@@ -0,0 +1,44 @@
/**
* Kettle Sample app - multipartForm client
*
* Copyright 2018 OCAD University
*
* Licensed under the New BSD license. You may not use this file except in
* compliance with this License.
*
* You may obtain a copy of the License at
* https://github.com/gpii/universal/LICENSE.txt
*/

"use strict";

var fluid = require("infusion"),
kettle = require("../../kettle.js"),
examples = fluid.registerNamespace("examples");

/* Client definitions - issue a request against the Kettle config's server */

// Gain access to kettle.test.request definitions
kettle.loadTestingSupportQuiet();

// Define a test component firing a request to the server
var request = kettle.test.request.formData({
path: "/upload",
method: "POST",
formData: {
files: {
image: "@expand:fluid.module.resolvePath(%kettle/tests/data/multer/test.png)"
}
},
listeners: {
onComplete: "examples.uploadConfig.receiver"
}
});

examples.uploadConfig.receiver = function (body) {
console.log("Successfully received response ", body, " from Kettle server");
process.exit(0);
};

// Send the POST request to the server
request.send();
90 changes: 90 additions & 0 deletions examples/multiPartForm/multiPartForm-code.js
@@ -0,0 +1,90 @@
/**
* Kettle Sample app - multipartForm using code
*
* Copyright 2018 OCAD University
*
* Licensed under the New BSD license. You may not use this file except in
* compliance with this License.
*
* You may obtain a copy of the License at
* https://github.com/gpii/universal/LICENSE.txt
*/

"use strict";

var fluid = require("infusion"),
examples = fluid.registerNamespace("examples");

require("../../kettle.js");

fluid.defaults("examples.uploadConfig", {
"gradeNames": ["fluid.component"],
"components": {
"server": {
"type": "kettle.server",
"options": {
"port": 8081,
"components": {
"imageUpload": {
"type": "kettle.middleware.multer",
"options": {
"formFieldOptions": {
"method": "single",
"fieldName": "image"
},
"members": {
"storage": "{that}.diskStorage"
},
"invokers": {
"diskStorageDestination": {
"funcName": "examples.uploadConfig.diskStorageDestination"
}
}
}
},
"app": {
"type": "kettle.app",
"options": {
"requestHandlers": {
"imageUploadHandler": {
"type": "examples.uploadConfig.handler",
"route": "/upload",
"method": "post"
}
}
}
}
}
}
}
}
});

examples.uploadConfig.diskStorageDestination = function (req, file, cb) {
cb(null, "./examples/multipartForm/uploads");
};

fluid.defaults("examples.uploadConfig.handler", {
gradeNames: "kettle.request.http",
requestMiddleware: {
imageUpload: {
middleware: "{server}.imageUpload"
}
},
invokers: {
handleRequest: "examples.uploadConfig.handleRequest"
}
});

examples.uploadConfig.handleRequest = function (request) {
var uploadedFileDetails = request.req.file;
request.events.onSuccess.fire({
message: fluid.stringTemplate("POST request received on path /upload; file %originalName uploaded to %uploadedPath", {originalName: uploadedFileDetails.originalname, uploadedPath: uploadedFileDetails.path})
});
};

// Construct the server using the above config
examples.uploadConfig();

// Load the client to make a test request
require("./multipartForm-client.js");
Empty file.
4 changes: 3 additions & 1 deletion kettle.js
@@ -1,7 +1,7 @@
/*
Kettle Main Module Loader

Copyright 2012-2013 OCAD University
Copyright 2012-2018 OCAD University
Copyright 2015 Raising the Floor (International)

Licensed under the New BSD license. You may not use this file except in
Expand All @@ -25,6 +25,7 @@ require("./lib/KettleApp.js");
require("./lib/KettleConfigLoader.js");
require("./lib/KettleResolvers.js");
require("./lib/KettleMiddleware.js");
require("./lib/KettleMiddleware-Multer.js");
require("./lib/KettleMultiConfig.js");
require("./lib/KettleRouter.js");
require("./lib/KettleRequest.js");
Expand All @@ -36,6 +37,7 @@ require("./lib/KettleSession.js");
kettle.loadTestingSupport = function () {
require("./lib/test/KettleTestUtils.js");
require("./lib/test/KettleTestUtils.http.js");
require("./lib/test/KettleTestUtils.form.js");
require("./lib/test/KettleTestUtils.ws.js");
};

Expand Down