Define API/Spec for batch uploading sketches

[catarak]

Nature of issue?

• New feature request

New feature details:

Define endpoints so that users can batch upload sketches. This also includes the data structure of the sketches themselves. It's a little complicated right now given recursive file structures of folders and this involves generating ID's for file objects. It would be cool if users didn't have to do that and it happened magically on the backend.

[Vertmo]

Is the API being worked on right now ? That's seems like a bit of work, so it should probably be split in several parts; I'll start looking into it anyway ^^

[catarak]

You're right, this is a huge issue! I think a good first step would be dividing this into smaller issues, figuring out all of the steps that would need to go into this. We can discuss on this issue!

[Vertmo]

Ok, so the first thing would be to allow registered users to add API keys via their user settings, and of course store them in the database.
Then we will need to define secure authentification for the API (maybe we should use an already existing library).
We could then define two endpoints :

• /<username>/upload to upload a single sketch
• /<username>/batch-upload to upload several
  Of course, only would be able to successfully upload a sketch to their account.
  OK, that's only my first thoughts. Then we will need to work out the format of the upload request (especially how to upload a whole folder). But I'll need to study the existing code more thoroughly before that !
  In the meantime, can you tell me what you think of these first ideas ?

[catarak]

The first part is covered in #541!

My starting point for figuring out what API endpoints should be is, surprisingly, a fairly old Rails guide. Therefore, I think the endpoint for uploading a new sketch should be POST /<username/sketches. Which is the same as what the editor currently uses. Maybe for multiple sketches, it could be the same endpoint, but the API could check if the data is an array vs a singular project.

Also, to list some other stuff I think needs to happen:

• right now, the Node server that serves the client application, and the API are smushed together. I think that part of this work would be to separate them into two separate Node applications. We could even think about creating two different Docker images, and change the Kubernetes deployment to handle this.
• right now, when POSTing to the sketch#create route, it assumes the sketch has already been Mongo structure-ified. The confusing part of this is the files structure, which is a tree structure. I followed a Mongo Tree Structure Guide to create a data structure in which each file has an array of its children, and the children array contains ONLY the ids, not the full files. Maybe it doesn't make sense to expect users to do this manually. I wonder what is a realistic data structure for users! Maybe just a nested object/array for files, which gets Mongo-ified server-side?

[Vertmo]

• Ok, since it doesn't seem anyone is directly working on API authentication with Token (or oauth?) #541, I'll probably start with that this week-end ^^
• You're right, it seems more logical to avoid having too many different endpoints.
• Are you sure separating application and API is wise ? I don't really see the motivations (but I do see the complications). Could you elaborate on why you would want that ?
• Indeed, user shouldn't have to expect the internals of the application. I think a tree-like structure, but containing full files would be simpler to understand. Or maybe the simplest way for them (and us) would be to ask them to zip the sketch, so that the request is fairly simple to write ? (the problem could be the file limit for POST requests, since sketches can potentially include lots of images and sounds)

[catarak]

• Not sure if these reasons make sense, but this is what I was thinking about with respect to separating the front end and the API:

  • Code-cleanliness/organization. It would be kind of nice to have one app that just handles rendering the front end, and one that does data stuff. It seems like that is what "real web applications" do. Maybe it's not broken and doesn't need to be fixed.
  • Deployment. It might be nice to be able to scale the components separately. Maybe this is over-engineering the problem. I am super new to dev ops and don't really know what I'm doing. Given that I have autoscaling on, and Kubernetes currently uses 1-2 pods, it doesn't seem super important.

  Now that I write this all out, it doesn't seem that important!

• I wonder what the best way to handle images and sounds would be! They're stored on S3, so the API would have to handle uploading the files to S3 and then storing the url with the sketch files. Maybe a zip would make sense. Maybe the POST /<username>/sketches route can't handle images/sounds, but a separate route can handle zip files. There are a lot of options!

[Vertmo]

• Well I know nothing at all about DevOps, so your guess is probably better than mine !
• I'll do a bit of research about all of that, and I'll come back to you as soon as I have a good idea of how we can do things.

Also ! (and this is kind of unrelated to the issue, but I don't know where to talk about this) I'll be hosting a Hacktoberfest event Saturday, so I'll try to send students this way (and more generally to the processing repositories).