Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix file upload - consider metadata options by creating form data #500

Merged
merged 30 commits into from
Mar 21, 2018
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
e58136b
update node/npm and reinstall packages
aoberoi Mar 14, 2018
5feaa6b
add unit tests for WebClient without a token
aoberoi Mar 14, 2018
e6580bb
adds typings-tester, and failing test for WebClient without token
aoberoi Mar 14, 2018
70b2d74
lintstaged ignores test typescript
aoberoi Mar 14, 2018
99db715
allow WebClient token to be undefined, fixes #482
aoberoi Mar 14, 2018
93d8350
allow integration test cases to take up to 4s before timing out
aoberoi Mar 14, 2018
73dea90
fix file upload - consider metadata options by creating form data, re…
KharitonOff Mar 14, 2018
8e4d4ee
fix unregular build failure
KharitonOff Mar 14, 2018
9d0801c
remove nyc from integration tests, it broke codecov
aoberoi Mar 14, 2018
805dd6f
Merge pull request #499 from aoberoi/webclient-no-token
aoberoi Mar 14, 2018
5ce84bb
exports method argument types, fixes #483
aoberoi Mar 14, 2018
dee4b03
Merge pull request #502 from aoberoi/export-method-argument-types
aoberoi Mar 14, 2018
7f8e5b7
more tightly specifies OAuthAccessArguments and OAuthTokenArguments, …
aoberoi Mar 14, 2018
8e4da08
removes pjson and directly requires package.json, fixes #478
aoberoi Mar 14, 2018
945439a
Merge pull request #504 from aoberoi/remove-pjson
aoberoi Mar 14, 2018
501b0af
update for boolean flag quirk
aoberoi Mar 14, 2018
6902c4e
Merge pull request #503 from aoberoi/oauth-argument-types
aoberoi Mar 14, 2018
8ce3b61
v4.0.1
aoberoi Mar 14, 2018
a206c3e
Merge pull request #506 from slackapi/rel-v4.0.1
aoberoi Mar 15, 2018
e96b059
add assertion for filename
KharitonOff Mar 15, 2018
ba661a7
put forEach statement in one line
KharitonOff Mar 15, 2018
45c7702
fix file upload - consider metadata options by creating form data, re…
KharitonOff Mar 14, 2018
c1d65b5
fix unregular build failure
KharitonOff Mar 14, 2018
bb58541
add assertion for filename
KharitonOff Mar 15, 2018
e7bbd38
put forEach statement in one line
KharitonOff Mar 15, 2018
f5596ba
resolve merge conflict
KharitonOff Mar 15, 2018
b0a7d94
resolve merge conflict
KharitonOff Mar 15, 2018
1899050
increase test timeout in order to avoid unexpected build failure
KharitonOff Mar 15, 2018
6de72b3
increase mocha tests timeout to 10 sec
KharitonOff Mar 16, 2018
7c89cac
remove check for options in file attribute, remove unit test
KharitonOff Mar 20, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"typescript.tsdk": "./node_modules/typescript/lib",
"editor.rulers": [
120
]
}
"typescript.tsdk": "./node_modules/typescript/lib",
"editor.rulers": [
120
],
"editor.tabSize": 2,
}
2 changes: 1 addition & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

100 changes: 56 additions & 44 deletions src/WebClient.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,12 @@ describe('WebClient', function () {
});
});

describe('apiCall()', function() {
describe('apiCall()', function () {
beforeEach(function () {
this.client = new WebClient(token, { retryConfig: fastRetriesForTest });
});

describe('when making a successful call', function() {
describe('when making a successful call', function () {
beforeEach(function () {
this.scope = nock('https://slack.com')
.post(/api/)
Expand Down Expand Up @@ -149,9 +149,9 @@ describe('WebClient', function () {
hum: false,
},
})
.then(() => {
scope.done();
});
.then(() => {
scope.done();
});
});

it.skip('should remove undefined or null values from API arguments');
Expand Down Expand Up @@ -201,18 +201,28 @@ describe('WebClient', function () {

// intentially vague about the method name
return this.client.apiCall('upload', {
file: imageBuffer,
filename: 'train.png',
})
file: imageBuffer,
filename: 'train.png',
})
.then((parts) => {
assert.lengthOf(parts.files, 1);
const file = parts.files[0];
// options were not provided to the form builder
assert.include(file, { fieldname: 'file' });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still not asserting that the filename field exists, and removed the comment that would remind us to improve the tests so that it is asserted. why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, I've overseen this assertion statement and felt that the fix solved the problem pointed in the comment. I'll bring your TODO and assertion back.

});
});

it('should properly serialize when the file is an object with value as a Buffer and options containing filename', function () {
const imageBuffer = fs.readFileSync(path.resolve('test', 'fixtures', 'train.jpg'));

// intentially vague about the method name
return this.client.apiCall('upload', {
file: { value: imageBuffer, options: { filename: 'train.png' } },
filename: 'train.png',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was trying to avoid this usage all together, because it seems redundant to have to specify the filename twice. can you see any advantage to doing it this way?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main advantage I see is the current implementation state of all the folks who used the old style. I don't know whether there is a need for someone to provide further attributes in the options field (e.g. contentType or knownLength)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i also want to make sure the implementation accounts for those who used the old style, but i prefer that the path we take to get there is not to redo the same hack, but to smooth out the API so that you don't need to redundantly specify the filename.

i wouldn't call it a hack if there was legitimately a use case or need for using the { value, options } format for file. i don't see any benefit from exposing contentType and/or knownLength at this time, since that only has the potential to force to incorrect values and cause the upload to fail (but I'm happy to be corrected if there is a real use case).

maybe we can channel this effort towards investigating why the following code didn't behave as we want:

// options.file = {
// // @ts-ignore
// value: options.file,
// options: {
// // @ts-ignore
// filename: options.filename,
// },
// };

AFAICT it should result in the same thing...

also, have you tried this implementation to upload actual files? i'm not 100% confident that the nock expectations are correct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but I don't really understand your comment... this code snippet, you mentioned above works now as well. Off course I have tried it before I've opened this PR. And the problem with it, why it didn't work before was in the implementation of

if (containsBinaryData) {
return flattened.reduce((form, [key, value]) => {
form.append(key, value);
return form;
}, new FormData());
}

I've added this part in order to provide the attribute options to the form builder.

if (key === 'file' && value.value) {
    form.append(key, value.value, value.options);
    return form;
}

https://github.com/KharitonOff/node-slack-sdk/blob/45c77022a81cd2631a5a2bd3ee0dd29e401f51d7/src/WebClient.ts#L554-L563

This is actually the main goal of this PR and all other things we talked here about, were done with intention not to break the current state. As I mentioned in the PR description, you can upload files now with no need of additional/redundant information

let file = fs.readFileSync('../file.png')
slackClient.files.upload({ channels, file, filename: 'file.png' })

Should i remove the 'backup' solution to restrict the file upload to only one way?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@KharitonOff thanks so much for your clarification. i missed that diff amidst the rest of the changes but i now see where the issue was before.

yes, i'd prefer removing the backup solution so that we don't unnecessarily expand the API contract. if you can do that, i'm happy to land this PR!

})
.then((parts) => {
assert.lengthOf(parts.files, 1);
const file = parts.files[0];
// TODO: understand why this assertion is failing. already employed the buffer metadata workaround, should
// look into the details about whether that workaround is still required, or why else the `source.on` is not
// defined error would occur, or if Slack just doesn't need a filename for the part
// assert.include(file, { fieldname: 'file', filename: 'train.jpg' });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note to self: should the assertion with filename work now?

// NOTE: it seems the file and its filename are emitted as a field in addition to the token, not sure if
// this was happening in the old implementation.
assert.include(file, { fieldname: 'file' });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still not asserting that the filename field exists.

});
});
Expand All @@ -225,8 +235,8 @@ describe('WebClient', function () {

// intentially vague about the method name
return this.client.apiCall('upload', {
file: imageBuffer,
})
file: imageBuffer,
})
.then(() => {
const output = this.capture.getCapturedText();
assert.isNotEmpty(output);
Expand All @@ -241,8 +251,8 @@ describe('WebClient', function () {
const imageStream = fs.createReadStream(path.resolve('test', 'fixtures', 'train.jpg'));

return this.client.apiCall('upload', {
file: imageStream,
})
file: imageStream,
})
.then((parts) => {
assert.lengthOf(parts.files, 1);
const file = parts.files[0];
Expand All @@ -265,20 +275,20 @@ describe('WebClient', function () {
describe('metadata in the user agent', function () {
it('should set the user agent to contain package metadata', function () {
const scope = nock('https://slack.com', {
reqheaders: {
'User-Agent': (value) => {
const metadata = parseUserAgentIntoMetadata(value)
// NOTE: this assert isn't that strong and doesn't say anything about the values. at this time, there
// isn't a good way to test this without dupicating the logic of the code under test.
assert.containsAllKeys(metadata, ['node', '@slack:client']);
// NOTE: there's an assumption that if there's any keys besides these left at all, its the platform part
delete metadata.node;
delete metadata['@slack:client'];
assert.isNotEmpty(metadata);
return true;
},
reqheaders: {
'User-Agent': (value) => {
const metadata = parseUserAgentIntoMetadata(value)
// NOTE: this assert isn't that strong and doesn't say anything about the values. at this time, there
// isn't a good way to test this without dupicating the logic of the code under test.
assert.containsAllKeys(metadata, ['node', '@slack:client']);
// NOTE: there's an assumption that if there's any keys besides these left at all, its the platform part
delete metadata.node;
delete metadata['@slack:client'];
assert.isNotEmpty(metadata);
return true;
},
})
},
})
.post(/api/)
.reply(200, { ok: true });
return this.client.apiCall('method')
Expand All @@ -291,14 +301,14 @@ describe('WebClient', function () {
const [name, version] = ['appmedataname', 'appmetadataversion'];
addAppMetadata({ name, version });
const scope = nock('https://slack.com', {
reqheaders: {
'User-Agent': (value) => {
const metadata = parseUserAgentIntoMetadata(value)
assert.propertyVal(metadata, name, version);
return true;
},
reqheaders: {
'User-Agent': (value) => {
const metadata = parseUserAgentIntoMetadata(value)
assert.propertyVal(metadata, name, version);
return true;
},
})
},
})
.post(/api/)
.reply(200, { ok: true });
// NOTE: appMetaData is only evalued on client construction, so we cannot use the client already created
Expand Down Expand Up @@ -331,7 +341,7 @@ describe('WebClient', function () {
});
})

describe('has option to change slackApiUrl', function() {
describe('has option to change slackApiUrl', function () {
it('should send requests to an alternative URL', function () {
const alternativeUrl = 'http://12.34.56.78/api/';
const scope = nock(alternativeUrl)
Expand All @@ -344,7 +354,7 @@ describe('WebClient', function () {

describe('has an option to set a custom HTTP agent', function () {
// not confident how to test this. one idea is to use sinon to intercept method calls on the agent.
it.skip('should send a request using the custom agent', function() {
it.skip('should send a request using the custom agent', function () {
const agent = new Agent();
const client = new WebClient(token, { agent });
return client.apiCall('method');
Expand Down Expand Up @@ -390,7 +400,7 @@ describe('WebClient', function () {
// verify that any requests after maxRequestConcurrency were delayed by the responseDelay
const queuedResponses = responses.slice(3);
const minDiff = concurrentResponses[concurrentResponses.length - 1].diff + responseDelay;
queuedResponses.forEach(r => assert.isAbove(r.diff, minDiff));
queuedResponses.forEach(r => assert.isAtLeast(r.diff, minDiff));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was this actually causing a failure? i think your change is more technically correct, but i'm wondering if you actually got a test run where the timers were so perfect that r.diff and minDiff were actually equal.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that I've run into the issue of getting a test run where the timers matched up and a false positive was thrown--just once, but it has happened.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I added this fix to the PR, because the build of my first commit broke accidentally

});
});

Expand All @@ -403,13 +413,15 @@ describe('WebClient', function () {
assert.lengthOf(responses, 2);

// verify that maxRequestConcurrency requets were all sent concurrently
const concurrentResponses = responses.slice(0, 1); // the first 3 responses
const concurrentResponses = responses.slice(0, 1); // the first response
concurrentResponses.forEach(r => assert.isBelow(r.diff, responseDelay));

// verify that any requests after maxRequestConcurrency were delayed by the responseDelay
const queuedResponses = responses.slice(1);
const queuedResponses = responses.slice(1);// the second response
const minDiff = concurrentResponses[concurrentResponses.length - 1].diff + responseDelay;
queuedResponses.forEach(r => assert.isAbove(r.diff, minDiff));
queuedResponses.forEach(r => {
assert.isAtLeast(r.diff, minDiff)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this could fit in a single line rather than a block.

});
});
});

Expand Down
28 changes: 15 additions & 13 deletions src/WebClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -524,18 +524,16 @@ export class WebClient extends EventEmitter {
// 'fail. add the `filename` option to fix.');
// }

// Buffers are sometimes not handled well by the underlying `form-data` package. Adding extra metadata resolves
// that issue. See: https://github.com/slackapi/node-slack-sdk/issues/307#issuecomment-289231737
// @ts-ignore
// options.file = {
// // @ts-ignore
// value: options.file,
// options: {
// // @ts-ignore
// filename: options.filename,
// },
// };
// }
// Buffers are sometimes not handled well by the underlying `form-data` package. Adding extra metadata resolves
// that issue. See: https://github.com/slackapi/node-slack-sdk/issues/307#issuecomment-289231737
if (options['file'] && !options['file'].value) {
options['file'] = {
value: options['file'],
options: {
filename: options['filename'],
},
};
}
}

const flattened = objectEntries(options)
Expand All @@ -555,6 +553,10 @@ export class WebClient extends EventEmitter {
// a body with binary data should be serialized as multipart/form-data
if (containsBinaryData) {
return flattened.reduce((form, [key, value]) => {
if (key === 'file' && value.value) {
form.append(key, value.value, value.options);
return form;
}
form.append(key, value);
return form;
}, new FormData());
Expand Down Expand Up @@ -658,7 +660,7 @@ interface FormCanBeURLEncoded {
[key: string]: string | number | boolean;
}

interface BodyCanBeFormMultipart extends Readable {}
interface BodyCanBeFormMultipart extends Readable { }

/**
* Determines whether a request body object should be treated as FormData-encodable (Content-Type=multipart/form-data).
Expand Down