-
Notifications
You must be signed in to change notification settings - Fork 948
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Summary: Changelog: [Network] Non-binary request are not properly utf-8 decoded on both iOS and Android, both when gzipped and when not gzipped This diff fixes a long standing / ping-pong issue regarding network decoding differences between * iOS vs Android * binary vs utf-8 * gzipped vs uncompressed The changes aren't too big, but the underlying investigating is :) The primary contribution to this diff is: First, adding test cases for know problematic cases. This is done by grabbing the messages that are send from the flipper client to flipper using the flipper messages plugin. This is the base64 data that is stored in the `.txt` files. Beyond that, for all tests public endpoints are used, so that we can both get a hold of the raw original files, and how we expect them to be displayed in flipper. For testing a simple RN app was build, with a button that fires a bunch requests. The first 3 are captured in unit tests, the last one is not idempotent, but a case reported in #1466, so just left it there as manual verification. ``` const fetchData = async () => { await fetch( 'https://raw.githubusercontent.com/SangKa/MobX-Docs-CN/master/docs/donating.md', { headers: { 'Accept-Encoding': 'identity', // signals that we don't want gzip }, }, ); await fetch('https://reactnative.dev/img/tiny_logo.png?x=' + Math.random()); await fetch( 'https://raw.githubusercontent.com/SangKa/MobX-Docs-CN/master/docs/donating.md', ); await fetch( 'https://ex.ke.com/sdk/recommend/html/100001314?hdicCityId=110000¶mMap[source]=&id=100001314&mediumId=100000037&elementId=&resblockId=1111027381003&templateConfig=%5Bobject%20Object%5D&fbExpoId=346620976471638017&fbQueryId=&required400=true&unique=1111027381003&parentSceneId=', ); }; ``` The second contribution of this diff is that it doesn't use weird URLencoder hacks to convert base64 to utf8, but rather a proper library. The problem with our original solution, using `atob` is that it converts to ASCII, not to utf-8, which is the source of the original bugs. See for more background on this: https://www.npmjs.com/package/js-base64#decode-vs-atob-and-encode-vs-btoa- Solves: #1466 #1541 #1458 Supersedes D23837750 Future work: we don't inspect the `content-type=xxx;charset` header yet, which we should do for less common encodings, to make sure that they get displayed correctly as well Future work: in feature like copy data and curl, we always call decode body, without check if we are actually dealing with non-binary data. Probably it is better to keep binary data in base64, rather than decoding it, as that will assume the data is an utf-8 string, which might fail. An assumption in these changes is that binary data is never gzipped, which is generally correct; gzip is not applied by webserver to things like images, as it would increase, not decrease their size, and waste a lot of computation power. Reviewed By: cekkaewnumchai Differential Revision: D23403095 fbshipit-source-id: 5099cc4a7503f0f63bd10585dc6590ba893f3dde
- Loading branch information
1 parent
5c82b9d
commit 6b7b1fa
Showing
15 changed files
with
143 additions
and
32 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
/** | ||
* Copyright (c) Facebook, Inc. and its affiliates. | ||
* | ||
* This source code is licensed under the MIT license found in the | ||
* LICENSE file in the root directory of this source tree. | ||
* | ||
* @format | ||
*/ | ||
|
||
import {readFile} from 'fs'; | ||
import path from 'path'; | ||
import {decodeBody} from '../utils'; | ||
import {Response} from '../types'; | ||
import {promisify} from 'util'; | ||
import {readFileSync} from 'fs'; | ||
|
||
async function createMockResponse(input: string): Promise<Response> { | ||
const inputData = await promisify(readFile)( | ||
path.join(__dirname, 'fixtures', input), | ||
'ascii', | ||
); | ||
const gzip = input.includes('gzip'); // if gzip in filename, assume it is a gzipped body | ||
const testResponse: Response = { | ||
id: '0', | ||
timestamp: 0, | ||
status: 200, | ||
reason: 'dunno', | ||
headers: gzip | ||
? [ | ||
{ | ||
key: 'Content-Encoding', | ||
value: 'gzip', | ||
}, | ||
] | ||
: [], | ||
data: inputData.replace(/\s+?/g, '').trim(), // remove whitespace caused by copy past of the base64 data, | ||
isMock: false, | ||
insights: undefined, | ||
totalChunks: 1, | ||
index: 0, | ||
}; | ||
return testResponse; | ||
} | ||
|
||
describe('network data encoding', () => { | ||
const donatingExpected = readFileSync( | ||
path.join(__dirname, 'fixtures', 'donating.md'), | ||
'utf-8', | ||
).trim(); | ||
const tinyLogoExpected = readFileSync( | ||
path.join(__dirname, 'fixtures', 'tiny_logo.png'), | ||
); | ||
const tinyLogoBase64Expected = readFileSync( | ||
path.join(__dirname, 'fixtures', 'tiny_logo.base64.txt'), | ||
'utf-8', | ||
); | ||
|
||
test('donating.md.utf8.ios.txt', async () => { | ||
const response = await createMockResponse('donating.md.utf8.ios.txt'); | ||
expect(decodeBody(response).trim()).toEqual(donatingExpected); | ||
}); | ||
|
||
test('donating.md.utf8.gzip.ios.txt', async () => { | ||
const response = await createMockResponse('donating.md.utf8.gzip.ios.txt'); | ||
expect(decodeBody(response).trim()).toEqual(donatingExpected); | ||
}); | ||
|
||
test('donating.md.utf8.android.txt', async () => { | ||
const response = await createMockResponse('donating.md.utf8.android.txt'); | ||
expect(decodeBody(response).trim()).toEqual(donatingExpected); | ||
}); | ||
|
||
test('donating.md.utf8.gzip.android.txt', async () => { | ||
const response = await createMockResponse( | ||
'donating.md.utf8.gzip.android.txt', | ||
); | ||
expect(decodeBody(response).trim()).toEqual(donatingExpected); | ||
}); | ||
|
||
test('tiny_logo.android.txt', async () => { | ||
const response = await createMockResponse('tiny_logo.android.txt'); | ||
expect(response.data).toEqual(tinyLogoExpected.toString('base64')); | ||
}); | ||
|
||
test('tiny_logo.android.txt - encoded', async () => { | ||
const response = await createMockResponse('tiny_logo.android.txt'); | ||
// this compares to the correct base64 encoded src tag of the img in Flipper UI | ||
expect(response.data).toEqual(tinyLogoBase64Expected.trim()); | ||
}); | ||
|
||
test('tiny_logo.ios.txt', async () => { | ||
const response = await createMockResponse('tiny_logo.ios.txt'); | ||
expect(response.data).toEqual(tinyLogoExpected.toString('base64')); | ||
}); | ||
|
||
test('tiny_logo.ios.txt - encoded', async () => { | ||
const response = await createMockResponse('tiny_logo.ios.txt'); | ||
// this compares to the correct base64 encoded src tag of the img in Flipper UI | ||
expect(response.data).toEqual(tinyLogoBase64Expected.trim()); | ||
}); | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# 捐赠 | ||
|
||
MobX 是使您的项目成功的关键吗? 使用[捐赠按钮](https://mobxjs.github.io/mobx/donate.html)分享胜利!如果你留下一个名字,它将被添加到赞助商列表。 |
1 change: 1 addition & 0 deletions
1
desktop/plugins/network/__tests__/fixtures/donating.md.utf8.android.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
IyDmjZDotaAKCk1vYlgg5piv5L2/5oKo55qE6aG555uu5oiQ5Yqf55qE5YWz6ZSu5ZCX77yfIOS9 v+eUqFvmjZDotaDmjInpkq5dKGh0dHBzOi8vbW9ieGpzLmdpdGh1Yi5pby9tb2J4L2RvbmF0ZS5o dG1sKeWIhuS6q+iDnOWIqe+8geWmguaenOS9oOeVmeS4i+S4gOS4quWQjeWtl++8jOWug+Wwhuii q+a3u+WKoOWIsOi1nuWKqeWVhuWIl+ihqOOAggo= |
1 change: 1 addition & 0 deletions
1
desktop/plugins/network/__tests__/fixtures/donating.md.utf8.gzip.android.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
H4sIAAAAAAAAAyWNXQvBUByH7/cpVm642e59B/dKLiwywpSjXDKsY6gl8hrjQkNGSeYtH8b5n51d +QoWl7+n5+kX4GnXYGeT4yKKFOXp6ECeL6pa7qThLa/u1KbYAH3hT2ievL4NxvDzWPC+5Pat2L+l nZbXs+NBGaFiKSyKeUWqZEtCOoPksiRklB8Qk0ohgVKCjPK5EGCN3HasPgO8+TxqsFbpfEaepjsY E6dNnCpxtmB0Ye+fdcCuw1Fjqx293EE3AR/ZeQ76BgYa4CFbWu+qyn0Bz88iqcgAAAA= |
1 change: 1 addition & 0 deletions
1
desktop/plugins/network/__tests__/fixtures/donating.md.utf8.gzip.ios.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
IyDmjZDotaAKCk1vYlgg5piv5L2/5oKo55qE6aG555uu5oiQ5Yqf55qE5YWz6ZSu5ZCX77yfIOS9v+eUqFvmjZDotaDmjInpkq5dKGh0dHBzOi8vbW9ieGpzLmdpdGh1Yi5pby9tb2J4L2RvbmF0ZS5odG1sKeWIhuS6q+iDnOWIqe+8geWmguaenOS9oOeVmeS4i+S4gOS4quWQjeWtl++8jOWug+Wwhuiiq+a3u+WKoOWIsOi1nuWKqeWVhuWIl+ihqOOAggo= |
1 change: 1 addition & 0 deletions
1
desktop/plugins/network/__tests__/fixtures/donating.md.utf8.ios.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
IyDmjZDotaAKCk1vYlgg5piv5L2/5oKo55qE6aG555uu5oiQ5Yqf55qE5YWz6ZSu5ZCX77yfIOS9v+eUqFvmjZDotaDmjInpkq5dKGh0dHBzOi8vbW9ieGpzLmdpdGh1Yi5pby9tb2J4L2RvbmF0ZS5odG1sKeWIhuS6q+iDnOWIqe+8geWmguaenOS9oOeVmeS4i+S4gOS4quWQjeWtl++8jOWug+Wwhuiiq+a3u+WKoOWIsOi1nuWKqeWVhuWIl+ihqOOAggo= |
1 change: 1 addition & 0 deletions
1
desktop/plugins/network/__tests__/fixtures/tiny_logo.android.txt
Large diffs are not rendered by default.
Oops, something went wrong.
1 change: 1 addition & 0 deletions
1
desktop/plugins/network/__tests__/fixtures/tiny_logo.base64.txt
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters