Multi File Upload component for Vue.js to create an initial JSON or XML database

[JohannesLichtenberger]

No description provided.

[JohannesLichtenberger]

Repository: https://github.com/sirixdb/sirix-web-frontend

[ghost]

Hi, I would take on this issue. Are there any custom validaitons you would like to have implemented?

[JohannesLichtenberger]

Hi Martin, hmm. I don't think so as you first will have to authenticate yourself to be able to upload something.

It's a bit tricky to get the HTTP-server up and running together with Keycloak, but someone fixed the Docker and docker-compose scripts. I think what's missing is to include the test realm configuration in the docker-compose file. Then ideally you can run it and send requests.

What you need to do first is: Doing a PUT HTTP-request to https://localhost:9433/database whereas the path parameter database is a user specified name for the database. Then you can do a POST request to the same URL whereas the file upload then is handled by Vert.x. You can then upload multiple files at once and the server generates the names for the resources in the database.

With both requests you have to provide the content-type (application/xml or application/json) as well as a a bearer token authorization HTTP-header:

https://sirix.io/rest-api.html

[JohannesLichtenberger]

I'd like to change the authorization OAuth2 flow for the web frontend, but so far you would have to use the password flow.

[JohannesLichtenberger]

Maybe you can use

https://element.eleme.io/#/en-US/component/upload

[ghost]

Yes I did use mentioned element, but came to a few issues here: If you want to have multiple files upload option with corresponding application/xml / application/json content-type. I will have to read file contents on frontend and merge json / xml files somehow, which makes no sense for me since creating initial database should be from one file only (just in my opinion). Another problem edge case would be, if user uploaded one XML and one JSON file. Since I can send only 1 request, there is no way to have 2 different content-types. I have no idea how is backend implemented but if you want to have multiple files upload, I suggest sending all files at once with content-type multipart/form-data. In that way, I can send you all the files and you decide what and how you want to process them. This is also more secure option, since data sent via frontend can be tampered with.

[JohannesLichtenberger]

Yeah, I saw this earlier as I have no integration test for this. I guess I need a handler in front to handle multipart/form-data and then split at the first encountered content-typd of each file (and all files must either be XML files or JSON files). What do you think? I might find a bit of time tomorrow to fix this.

[ghost]

Well, could you explain why you think multiple files are needed? I think that the initial DB should be created from 1 xml or json since I can't imagine, what could merging process possibly look like. As I have already mentioned before, as for security, I suggest processing all the data at the backend side and omit overhead of reading files on client.

[JohannesLichtenberger]

Yes, but a database might have a collection of resources. I thought it would be nice to simply upload a directory with a bunch of JSON or XML files into one XML or JSON database. So the database consists of multiple resources which somehow belong together :-)

I've changed the server code, but have to figure out how to best test this...

[ghost]

I see. The problem with uploading folder is its browser compatibility as it not included in HTML5 spec currently. We could use this, but it is not recommended for production usage. We can however allow uploading multiple files at once. The main question though remains: What content-type is the best to be used? Json / XML with additional reading overhead on client (will need some merging strategy from you) or I can send files as they are via multipart/form-data with your processing on server.

[JohannesLichtenberger]

It should be multipart/form-data and then there are also content types for each file itself within the multipart stuff, right (explicitly allowed at least, see the link below)? So for each XML file I expect application/xml, for each JSON I expect application/json. I would go for uploading multiple files at once (can you check that all of them are either XML or JSON) files in the front-end already? I have another check on the backend, just using the content-type within the multipart content-type.

https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html

[JohannesLichtenberger]

Oh, It's not about the multipart/form-data Content-Type in the link, sorry. Seems we might have to use an inline content-disposition type and check this for XML or JSON data

[ghost]

I am not sure if I follow you with content-disposition type. I can definitely check, whether all files are either json or xml, that's not a problem and afterwards send it via multipart/form-data all at once. What you will get is list of binary files on server. I am not sure about Java, but for example I have implemented similar functionality in .NET Core not long ago and the framework mapped corresponding binaries into its own api objects which made working with them a breeze. Maybe you can find a similar mechanism for Java as well?

[JohannesLichtenberger]

Vert.x simple stores the uploaded files in an upload directory on the server. With the API I'm then importing these files into SirixDB.

So, with the multipart content-type I know that a bunch of files might have to be handled (or Vert.x knows) and then I still have to know if they are XML or JSON files:

https://github.com/sirixdb/sirix/blob/master/bundles/sirix-rest-api/src/main/kotlin/org/sirix/rest/crud/CreateMultipleResources.kt

[ghost]

I see. That seems like the way to go. Maybe a little improvement for your code: You don't actually have to iterate over all the files and then check if the total size equals js/xml count. Instead, with each iteration check, the content type and if it is for instance xml and jsonCount != 0 then you can return error immediatelly :)

[JohannesLichtenberger]

Yeah, right, thanks ;-)

[JohannesLichtenberger]

Hi @MartinTuroci, still interested in this? I'd love to move forward, but I myself currently also have not much spare time, sadly.

Have a great weekend 👍

[ghost]

Hi @JohannesLichtenberger, unfortunately, I am still busy. I would love to look into this, but I cannot tell you when for sure. In case there is somebody else who is interested in this issue, go ahead. For the next 2 weeks at least, I will not be available.

[JohannesLichtenberger]

@MartinTuroci ah okay, no worries... maybe I'll find some time on my own :) but in any way, for testing I could also create a JSON and a XML database without the frontend 👍

[JohannesLichtenberger]

@MartinTuroci still no time? However #Hacktoberfest is over and maybe it was for the Hacktoberfest!?