Define API/Spec for batch uploading sketches

[catarak]

Nature of issue?

• New feature request

New feature details:

Define endpoints so that users can batch upload sketches. This also includes the data structure of the sketches themselves. It's a little complicated right now given recursive file structures of folders and this involves generating ID's for file objects. It would be cool if users didn't have to do that and it happened magically on the backend.

[Vertmo]

Is the API being worked on right now ? That's seems like a bit of work, so it should probably be split in several parts; I'll start looking into it anyway ^^

[catarak]

You're right, this is a huge issue! I think a good first step would be dividing this into smaller issues, figuring out all of the steps that would need to go into this. We can discuss on this issue!

[Vertmo]

Ok, so the first thing would be to allow registered users to add API keys via their user settings, and of course store them in the database.
Then we will need to define secure authentification for the API (maybe we should use an already existing library).
We could then define two endpoints :

• /<username>/upload to upload a single sketch
• /<username>/batch-upload to upload several
  Of course, only would be able to successfully upload a sketch to their account.
  OK, that's only my first thoughts. Then we will need to work out the format of the upload request (especially how to upload a whole folder). But I'll need to study the existing code more thoroughly before that !
  In the meantime, can you tell me what you think of these first ideas ?

[catarak]

The first part is covered in #541!

My starting point for figuring out what API endpoints should be is, surprisingly, a fairly old Rails guide. Therefore, I think the endpoint for uploading a new sketch should be POST /<username/sketches. Which is the same as what the editor currently uses. Maybe for multiple sketches, it could be the same endpoint, but the API could check if the data is an array vs a singular project.

Also, to list some other stuff I think needs to happen:

• right now, the Node server that serves the client application, and the API are smushed together. I think that part of this work would be to separate them into two separate Node applications. We could even think about creating two different Docker images, and change the Kubernetes deployment to handle this.
• right now, when POSTing to the sketch#create route, it assumes the sketch has already been Mongo structure-ified. The confusing part of this is the files structure, which is a tree structure. I followed a Mongo Tree Structure Guide to create a data structure in which each file has an array of its children, and the children array contains ONLY the ids, not the full files. Maybe it doesn't make sense to expect users to do this manually. I wonder what is a realistic data structure for users! Maybe just a nested object/array for files, which gets Mongo-ified server-side?

[Vertmo]

• Ok, since it doesn't seem anyone is directly working on API authentication with Token (or oauth?) #541, I'll probably start with that this week-end ^^
• You're right, it seems more logical to avoid having too many different endpoints.
• Are you sure separating application and API is wise ? I don't really see the motivations (but I do see the complications). Could you elaborate on why you would want that ?
• Indeed, user shouldn't have to expect the internals of the application. I think a tree-like structure, but containing full files would be simpler to understand. Or maybe the simplest way for them (and us) would be to ask them to zip the sketch, so that the request is fairly simple to write ? (the problem could be the file limit for POST requests, since sketches can potentially include lots of images and sounds)

[catarak]

• Not sure if these reasons make sense, but this is what I was thinking about with respect to separating the front end and the API:

  • Code-cleanliness/organization. It would be kind of nice to have one app that just handles rendering the front end, and one that does data stuff. It seems like that is what "real web applications" do. Maybe it's not broken and doesn't need to be fixed.
  • Deployment. It might be nice to be able to scale the components separately. Maybe this is over-engineering the problem. I am super new to dev ops and don't really know what I'm doing. Given that I have autoscaling on, and Kubernetes currently uses 1-2 pods, it doesn't seem super important.

  Now that I write this all out, it doesn't seem that important!

• I wonder what the best way to handle images and sounds would be! They're stored on S3, so the API would have to handle uploading the files to S3 and then storing the url with the sketch files. Maybe a zip would make sense. Maybe the POST /<username>/sketches route can't handle images/sounds, but a separate route can handle zip files. There are a lot of options!

[Vertmo]

• Well I know nothing at all about DevOps, so your guess is probably better than mine !
• I'll do a bit of research about all of that, and I'll come back to you as soon as I have a good idea of how we can do things.

Also ! (and this is kind of unrelated to the issue, but I don't know where to talk about this) I'll be hosting a Hacktoberfest event Saturday, so I'll try to send students this way (and more generally to the processing repositories).

[catarak]

i think a good reference could be @joeyklee's script to upload the ml5 examples to the web editor (https://github.com/processing/p5.js-web-editor/blob/master/server/scripts/examples-ml5.js). maybe the format of the sketches should be similar to how files are stored on github, and then once sent to the API, could be converted using the helper functions (or similar helper functions) that are in that uploader file.

[joeyklee]

@catarak - I'm happy to help in any way on this + very much looking forward to making this happen! Thank you!

[andrewn]

I've started thinking about the shape of the API that supports sketch uploads. #541 covers the authenticaton work to allow access to the API so the comments below assume it exists.

We already have a rich API that has sketch CRUD functionality, so the existing import scripts could be rewritten to use the existing API at /api.

The existing endpoints could also be extended to allow files to be created in a more friendly way. Specifically, IDs would be generated on the server instead of in the scripts (see #1073 for a proposal). This still means more code, but the ultimate aim could be updating the Editor client web app to use the new API fields.

The downside to this approach is that some of these endpoints would now become a public API and we'd need to be careful making changes to them. As part of this work, I'd like to write tests around whatever code is touched, so this would be a good way of making sure we don't break things going forward. And we'd document the endpoints that are now public.

Alternatively, we can create new endpoints that are the public API. This would reduce the implementation complexity, but would mean more code that would be doing very similar things.

Would be great to hear your thoughts @catarak.

[catarak]

As we talked about, I think the best way to move forward with this is to

1. Create a "dream" API specification (as you've already started working on!)
2. Create a middleware that checks if a user is authenticated with an access token
3. Figure out what of the current controllers can be reused/moved into functions

Then, we can write tests that will test both the public API and the API for the web editor!

[pelikhan]

You could maybe consider an api similar to codesandbox.io (https://codesandbox.io/docs/api). Easy way to generate a new sketch on the fly without having to pre-generate all examples in a script.