Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"image.jpg" is uploaded as a raw file and is not displayed in the browser. #3

Closed
parajbs opened this issue Nov 23, 2022 · 7 comments
Closed

Comments

@parajbs
Copy link

parajbs commented Nov 23, 2022

Hello redsolver and s5-dev team,

I have extensively tested the latest version 0.5.0 from the master.
The installation in Windows10 and Ubuntu was a little bit complicated and there were 4 small warnings, but it installed in both systems and it works.

Will test it today also on linux 32bit.
The previous problems with "p2p" version 0.3.0 are gone. awesome

But there are other problems, such as:

  • With a single upload of image.jpg or video.mp4 via "/s5/upload", the file is uploaded, but as type "RAW file" and not as type "image-file" or type "media-file ".
    If I want to call up the cids in the browser, the cid is downloaded and not displayed in the browser.

What am I doing wrong or is this a bug?

In the previous version 0.3.0 this worked great. In the new version 0.5.0 I can only display the individual files in the browser if I copy them to a directory before uploading and then upload the directory with the one file via "/s5/upload/directory".

I hope you can help.

@redsolver
Copy link
Collaborator

redsolver commented Nov 23, 2022

With version 0.5.0, I decided to remove the default "metadata file" structure. the reason for this is that before adding the new blake3/bao hash tree format, the metadata file was required for verified streaming of a large file in chunks. As part of this change, raw file CIDs now also contain the encoded file size. So with the new powerful raw CIDs that support both verified streaming and include the file size, the only advantage of having an additional metadata file would be filename and media type metadata which is pretty small for a single file. Adding this as a new metadata format and CID would however come with significantly lower performance for fetching files, because it would need two requests on the p2p network for every file.
So I decided to instead (optionally) make it possible to specify the file type as part of the CID too.
You can do this by just adding a normal file extension to the CID, for example z6e78acvH3M4PFaFzeA67UDpje2uYHai3LPSFQWs2WNinqpFzQYTa.mp4. (this example video does not contain audio)

This file can then for example be streamed in the browser using http://localhost:5522/z6e78acvH3M4PFaFzeA67UDpje2uYHai3LPSFQWs2WNinqpFzQYTa.mp4

Setting a custom download filename is not possible yet, but can be added easily if you have a good use case for it.

The new media file metadata format is meant to link multiple raw files that make up a single "media" item, for example different image formats and resolutions for the same image.

@redsolver
Copy link
Collaborator

Oh and thank you a lot for testing everything and your helpful feedback!

@parajbs
Copy link
Author

parajbs commented Nov 23, 2022

hello redsolver,

I'm very interested in the project and will help where I can.

I have already tried to integrate dnslink in version 0.3.0, since they had already integrated hns, but unfortunately this no longer works in version 0.5.0 either.
I saw in the code that you commented out some functions.

Being new to the Dart and Rust programming language myself, I'll have to wait for them to integrate it.
I'll test the code a bit.

Where I still have problems is to create a user, I don't understand yet.
And I still don't understand how to generate "Resolver CIDs" and "registy keys".

You should also expand "default_config.toml" with all valid entries, also commented out. To make it easier for other end users to find the right settings for their s5 node.
I only found them in code.

@parajbs
Copy link
Author

parajbs commented Nov 26, 2022

Hello @resolver

I despair right away. On all my 4 s5 nodes, I have the same problem when uploading.
If I upload "video.mp4" and then want to watch it in the browser, it doesn't work.

I have the test link
http://localhost:5522/z6e78acvH3M4PFaFzeA67UDpje2uYHai3LPSFQWs2WNinqpFzQYTa.mp4
tested by them and it works great too.
I downloaded and tested the testlink "testlink.mp4" and it was ok too.

Then I uploaded the "testlink.mp4" and tested the cid in the browser, and again it no longer works.
Then I downloaded the "defekte_testlink.mp4" and compared it with the "testlink.mp4".
When comparing the two files, I noticed that the sizes are different.
I then looked at the code of the files and noticed that there were 4 additional lines in the file "defekte_testlink.mp4".

----------------------------------------d4415fed0461cf06

Content disposition: form-data; name="jb1.mp4"; filename="jb1.mp4"

Content-Type: application/octet-stream


was pushed before the source code, making the file corrupt.
If I delete the 4 additional lines in the file "defekte_testlink.mp4", then the file works again when viewing.

I just saw that 4 additional lines were added to all uploaded files before the source code and 1 additional line was appended to the end of the file.

What's going on there?
Am I doing something wrong?
Or is that a bug in version 0.5.0?

Need help urgently as the problem is happening on all my 4 test s5 nodes.
Version 0.5.0 from the master is installed on all s5 nodes, and "store.local" is configured everywhere in the config.

i need help from you

@redsolver
Copy link
Collaborator

Because the new default raw files no longer need filename metadata, the upload endpoint is simplified to just upload the entire request body as file. So instead of sending a form request, try sending the entire file as the body without any additional metadata. How are you uploading the file?

@parajbs
Copy link
Author

parajbs commented Nov 26, 2022

single file:
curl -L -X POST "http://127.0.0.1:5522/s5/upload" -F "video1.mp4=@./video1.mp4"

for folders:
curl -L -X POST "http://127.0.0.1:5522/s5/upload/directory?filename=images" -F "images/image1.jpg=@./images/image1.jpg" -F "images/image2.jpg=@./images/image2.jpg"

or in uploads via my modified "nodejs-skynet" client via "scripts/upload.js" with the command:
node scripts/upload.js video1.mp4

@redsolver
Copy link
Collaborator

I think this is fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants