Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi File Upload component for Vue.js to create an initial JSON or XML database #81

Open
JohannesLichtenberger opened this issue Sep 26, 2019 · 20 comments

Comments

@JohannesLichtenberger
Copy link
Member

@JohannesLichtenberger JohannesLichtenberger commented Sep 26, 2019

No description provided.

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Sep 28, 2019

@MartinTuroci

This comment has been minimized.

Copy link

@MartinTuroci MartinTuroci commented Oct 4, 2019

Hi, I would take on this issue. Are there any custom validaitons you would like to have implemented?

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 4, 2019

Hi Martin, hmm. I don't think so as you first will have to authenticate yourself to be able to upload something.

It's a bit tricky to get the HTTP-server up and running together with Keycloak, but someone fixed the Docker and docker-compose scripts. I think what's missing is to include the test realm configuration in the docker-compose file. Then ideally you can run it and send requests.

What you need to do first is: Doing a PUT HTTP-request to https://localhost:9433/database whereas the path parameter database is a user specified name for the database. Then you can do a POST request to the same URL whereas the file upload then is handled by Vert.x. You can then upload multiple files at once and the server generates the names for the resources in the database.

With both requests you have to provide the content-type (application/xml or application/json) as well as a a bearer token authorization HTTP-header:

https://sirix.io/rest-api.html

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 4, 2019

I'd like to change the authorization OAuth2 flow for the web frontend, but so far you would have to use the password flow.

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 4, 2019

@MartinTuroci

This comment has been minimized.

Copy link

@MartinTuroci MartinTuroci commented Oct 4, 2019

Yes I did use mentioned element, but came to a few issues here: If you want to have multiple files upload option with corresponding application/xml / application/json content-type. I will have to read file contents on frontend and merge json / xml files somehow, which makes no sense for me since creating initial database should be from one file only (just in my opinion). Another problem edge case would be, if user uploaded one XML and one JSON file. Since I can send only 1 request, there is no way to have 2 different content-types. I have no idea how is backend implemented but if you want to have multiple files upload, I suggest sending all files at once with content-type multipart/form-data. In that way, I can send you all the files and you decide what and how you want to process them. This is also more secure option, since data sent via frontend can be tampered with.

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 4, 2019

Yeah, I saw this earlier as I have no integration test for this. I guess I need a handler in front to handle multipart/form-data and then split at the first encountered content-typd of each file (and all files must either be XML files or JSON files). What do you think? I might find a bit of time tomorrow to fix this.

@MartinTuroci

This comment has been minimized.

Copy link

@MartinTuroci MartinTuroci commented Oct 4, 2019

Well, could you explain why you think multiple files are needed? I think that the initial DB should be created from 1 xml or json since I can't imagine, what could merging process possibly look like. As I have already mentioned before, as for security, I suggest processing all the data at the backend side and omit overhead of reading files on client.

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 5, 2019

Yes, but a database might have a collection of resources. I thought it would be nice to simply upload a directory with a bunch of JSON or XML files into one XML or JSON database. So the database consists of multiple resources which somehow belong together :-)

I've changed the server code, but have to figure out how to best test this...

@MartinTuroci

This comment has been minimized.

Copy link

@MartinTuroci MartinTuroci commented Oct 5, 2019

I see. The problem with uploading folder is its browser compatibility as it not included in HTML5 spec currently. We could use this, but it is not recommended for production usage. We can however allow uploading multiple files at once. The main question though remains: What content-type is the best to be used? Json / XML with additional reading overhead on client (will need some merging strategy from you) or I can send files as they are via multipart/form-data with your processing on server.

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 5, 2019

It should be multipart/form-data and then there are also content types for each file itself within the multipart stuff, right (explicitly allowed at least, see the link below)? So for each XML file I expect application/xml, for each JSON I expect application/json. I would go for uploading multiple files at once (can you check that all of them are either XML or JSON) files in the front-end already? I have another check on the backend, just using the content-type within the multipart content-type.

https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 5, 2019

Oh, It's not about the multipart/form-data Content-Type in the link, sorry. Seems we might have to use an inline content-disposition type and check this for XML or JSON data

@MartinTuroci

This comment has been minimized.

Copy link

@MartinTuroci MartinTuroci commented Oct 5, 2019

I am not sure if I follow you with content-disposition type. I can definitely check, whether all files are either json or xml, that's not a problem and afterwards send it via multipart/form-data all at once. What you will get is list of binary files on server. I am not sure about Java, but for example I have implemented similar functionality in .NET Core not long ago and the framework mapped corresponding binaries into its own api objects which made working with them a breeze. Maybe you can find a similar mechanism for Java as well?

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 5, 2019

Vert.x simple stores the uploaded files in an upload directory on the server. With the API I'm then importing these files into SirixDB.

So, with the multipart content-type I know that a bunch of files might have to be handled (or Vert.x knows) and then I still have to know if they are XML or JSON files:

https://github.com/sirixdb/sirix/blob/master/bundles/sirix-rest-api/src/main/kotlin/org/sirix/rest/crud/CreateMultipleResources.kt

@MartinTuroci

This comment has been minimized.

Copy link

@MartinTuroci MartinTuroci commented Oct 5, 2019

I see. That seems like the way to go. Maybe a little improvement for your code: You don't actually have to iterate over all the files and then check if the total size equals js/xml count. Instead, with each iteration check, the content type and if it is for instance xml and jsonCount != 0 then you can return error immediatelly :)

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 5, 2019

Yeah, right, thanks ;-)

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 17, 2019

Hi @MartinTuroci, still interested in this? I'd love to move forward, but I myself currently also have not much spare time, sadly.

Have a great weekend 👍

@MartinTuroci

This comment has been minimized.

Copy link

@MartinTuroci MartinTuroci commented Oct 18, 2019

Hi @JohannesLichtenberger, unfortunately, I am still busy. I would love to look into this, but I cannot tell you when for sure. In case there is somebody else who is interested in this issue, go ahead. For the next 2 weeks at least, I will not be available.

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Oct 18, 2019

@MartinTuroci ah okay, no worries... maybe I'll find some time on my own :) but in any way, for testing I could also create a JSON and a XML database without the frontend 👍

@JohannesLichtenberger

This comment has been minimized.

Copy link
Member Author

@JohannesLichtenberger JohannesLichtenberger commented Nov 6, 2019

@MartinTuroci still no time? However #Hacktoberfest is over and maybe it was for the Hacktoberfest!?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.