reading binary file from fixture differs from drop event file

[BasieP]

Is this a Feature or Bug?

Bug

Current behavior:

When i get a binary file from a fixture and want to create a File() object (see: https://developer.mozilla.org/en-US/docs/Web/API/File) i get a wrong result (Some bad bytes)

When i drop a file in my application, and get the event.dataTransfer.Files[0] it is (slightly) different

Desired behavior:

I want an example how to convert a binary fixture to a File() object or a bugfix to make sure it is read correctly

How to reproduce:

I use a xlsx file (which is zip), add it too my fixtures and read it as i should.

Then catch the drop event of the same file and compare it.

Test code:

the normal dropcode:
https://codepen.io/anon/pen/GxzmqB

the test:

cy.fixture('test.xlsx', 'binary').then((myFile) => {
	const file = new File([myFile], 'test.xlsx', {
		type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
	})

	cy.get('myDropContainer').trigger('drop', {
		dataTransfer: {
			files: [file],
		},
	});
});

Testresults (with my file):

running the codepen + dropping manually:
80 75 3 4 20 0 6 0 8 0 0 0 33 0 113 14 57 43 112 1 0 0 160 5 0 0 19 0 219 1 91 67 111 110 116 101 110 116 95 84 121 112 101 115 93 46 120 109 108 32

running the codepen + 'dropping' with cypress:
80 75 3 4 20 0 6 0 8 0 0 0 33 0 113 14 57 43 112 1 0 0 194 160 5 0 0 19 0 195 155 1 91 67 111 110 116 101 110 116 95 84 121 112 101 115 93 46 120 109

• Cypress Version: 2.0.2
• Browser Version: latest chrome

[BasieP]

apart from that i kinda think it's an encoding issue and going over the options cypress has of reading fixtures it is kinda stunning to think they can read a file (binary) and represent it in a text string..

how the f* is that binary?

Go read this:
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

tldr:
if you want to give an option of reading a literal file (binary) then the return value should be a byte[]
not a string. cause a string ALWAYS has to have a character encoding. And binary is NOT and encoding.

ow and for that matter: how is base64 a character encoding, how is hex?
hex is a string representations of a char[] and base64 is an encoding of a string.. (not a file)

[jennifer-shehane]

Have you tried using cy.readFile() as a workaround to read in the .xlsx file?

[BasieP]

No i didn't but, i did it right now:

cy.readFile('cypress/fixtures/test.xlsx', 'binary').then((excel) {
})

throws an 'Error: the string "UTF-8 encode: second char code 0xe94d at index 434 in surrogate pair out of range" was thrown, throw an Error :)'
makes me wonder again... did i not state 'binary'? then why is it using utf8?

then again: seeing the 'encoding's' in the documentation of readfile being the same as those of fixture, i'm guessing the same internal method is used.

I have been testing a bit with the 'hex' option. Which looks like the only way to accually read the content of a binary file.
after converting the hex-string to a byte using the functions here:
https://gist.github.com/tauzen/3d18825ae41ff3fc8981
it DOES work.

I create a File object like this:

cy.fixture('test.xlsx', 'hex').then((excelHex) => {
    const excelBytes = hexStringToByte(excelHex);
    //create a File object
    const file = new File([excelBytes], 'test.xlsx', {
        type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
    });
    //and make the drop
    cy.get('canvas').trigger('drop', {
        dataTransfer: {
            files: [file],
        },
    });
});

[jennifer-shehane]

I looked into how we handle readFile and fixture, and it eventually calls Node's fs.readFile with the encoding here: https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/files.coffee#L15

[kammerer]

I run into a related issue when trying upload a shapefile. For uploads I am using cypress-file-upload.

1. Just using "binary" as encoding does not work for me. File upload results in InvalidCharacterError: Failed to execute 'atob' on 'Window': The string to be decoded is not correctly encoded. I specified encoding when reading fixture as well as for the payload to be uploaded.
2. The fix proposed by @BasieP does not work for me either - same error message.
3. Strangely, specifying encoding as "base64" does work.

[BasieP]

the bad thing is not in cypress but in the (very) crappy encoding and file read functions in nodejs itself.
fs.readfile has parameters that they call 'encoding' but those have nothing to do with actual character encodings.
Only text files need character encoding, binary files simply do not, cause they contain bits, not characters.

Nodejs fucked that up, and now it's a cascading failure...

[x-yuri]

There are probably two ways to handle it that come to mind:

    cy.fixture(fname).then(Cypress.Blob.base64StringToBlob).then(blob => {
        const file = new File([blob], fname, {type});

And:

    cy.fixture(fname, 'binary').then(content => {
        content = Uint8Array.from(content, x => x.charCodeAt(0))
        const file = new File([content], fname, {type});

I have a repository (branch) you can use to see it in action. There I was investigating different ways to upload a binary file using a traditional form (no AJAX, no React, no nothing).

And indeed I was thinking if Cypress should just say: "encoding is interpreted in the way nodejs does," or abstract away from that. And I think I lean towards the latter. The former sounds phpish.

[stijnherreman]

@jennifer-shehane

I looked into how we handle readFile and fixture, and it eventually calls Node's fs.readFile with the encoding here: https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/files.coffee#L15

Why does Cypress force an encoding? fs.readFile supports passing null for the encoding parameter so that you get the raw data, but Cypress falls back to 'utf8'. I don't know which of the two is used, but it occurs at https://github.com/cypress-io/cypress/blob/master/packages/server/lib/files.js#L9 and https://github.com/cypress-io/cypress/blob/master/packages/driver/src/cy/commands/files.ts#L18

It seems to be impossible to load the raw data of a fixture. I've tried methods from abramenal/cypress-file-upload#70, from this issue, and from various linked issues. But no matter what I try, either Cypress corrupts the data or throws an exception because of an encoding failure.
My data isn't text, encoding is irrelevant at best and harmful at worst. And no, 'binary' encoding does not help, it is merely a synonym for 'latin1' encoding.

[0xfacade]

Dear Cypress Team,

this issue is still present and affects everyone who works with binary files in their application. This is a pretty common use case, for example when users can upload an image, etc.

The issue could easily be fixed by setting the encoding passed to fs.readFile to null if the encoding passed to cy.fixture or cy.readFile is binary.

For all other frustrated users: if you want to upload a binary file using Cypress and the cypress-file-upload plugin, you can follow @x-yuri 's suggestion and read it as base64, then convert to binary again manually:

let files = [];                
filenames.map(filename => {
    cy.readFile('cypress/fixtures/' + filename, 'base64').then(contents => {
        files.push({
            fileContent: Cypress.Blob.base64StringToBlob(contents),
            fileName: filename,
            encoding: 'binary',
            mimeType: 'application/octet-stream'
        });
    });
});
cy.get('input[type=file]')
    .eq(0)
    .attachFile(files);

[cypress-bot]

The code for this is done in cypress-io/cypress#18534, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[cypress-bot]

Released in 9.0.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v9.0.0, please open a new issue.