New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attachments API #236

Closed
leplatrem opened this Issue Apr 28, 2015 · 8 comments

Comments

Projects
None yet
4 participants
@leplatrem
Contributor

leplatrem commented Apr 28, 2015

Along the storage and cache, there would be a attachments backend.

Uses-cases are:

  • Store big/files attached to your fxa account
  • Add metadata to files via the related record (tags, ...)
  • Query files using records API (filters, ...)

In the first iteration it should only:

  • allow to add/remove attachments on records
  • provide an internal api to allow more providers
  • persist files on the filesystem

Mimic of couchdb attachments sounds like a good start https://wiki.apache.org/couchdb/HTTP_Document_API#Attachments

In the long term, ideally it should:

@Natim

This comment has been minimized.

Contributor

Natim commented Sep 28, 2015

I was thinking a bit about this and here is a headsup:

http POST https://kinto.dev.mozaws.net/v1/buckets/default/collections/images/records/d0837177-0f92-cd01-42cb-4b712a206035/attachement/2015-09-28_10:59:08.jpg @2015-09-28_10:59:08.jpg

HTTP/1.1 201 Accepted
http POST https://kinto.dev.mozaws.net/v1/buckets/default/collections/images/records/d0837177-0f92-cd01-42cb-4b712a206035

{
    "attachements": {
        "2015-09-28_10:59:08.jpg": "https://eca95f6785b39.s3.amazonaws.com/2015-09-28_10:59:08.jpg"
    },
    "data": {
        "id": "d0837177-0f92-cd01-42cb-4b712a206035", 
        "last_modified": 1443430691318,
    }, 
    "permissions": {
        "write": [
            "basicauth:df93ca0ecaeaa3126595f6785b39c408be2539173c991a7b2e3181a9826a69bc"
        ]
    }
}

@MrChoclate

This comment has been minimized.

Contributor

MrChoclate commented Oct 30, 2015

Hello,

My use case will be to store tar archive.
Is there any work around waiting for this feature? Maybe something like storing raw bytes?

@leplatrem

This comment has been minimized.

Contributor

leplatrem commented Nov 2, 2015

@MrChoclate, we are working on a simple file attachment solution right now, probably as a plugin for a first version, but it won't be ready to use before the beginning of 2016 I guess.

What is your use-case exactly ? Did you build your own Cliquet-based application, or would it be through Kinto?

Meanwhile, as a work-around, depending on the file size you can:

  • Send the file content in a record field (as base64 for example)
  • Send the file somewhereelse (ex. Amazon S3) and store the URL in the record

Tell us what you think! Thanks for your interest in the project!

@MrChoclate

This comment has been minimized.

Contributor

MrChoclate commented Nov 2, 2015

@leplatrem I am building a Cliquet-based app. I need to do some server side operations and there is not current way to notify my server of changes in Kinto, right?

My use case is simple, a user send an archive containing source code. I extract it somewhere in the file system and compile source code (maybe use some kind of container to avoid security flaws) then store the result of the operation in the db. Of course this operations should be asynchronous because my user don't want to wait the compilation time to have a response.

I am not familiar with colander, will this simply work?

file = colander.SchemaNode(colander.String())
@Natim

This comment has been minimized.

Contributor

Natim commented Nov 2, 2015

there is not current way to notify my server of changes in Kinto, right?

Actually since Kinto 1.8.0 you can plug a notification listener that will tell you all about it in realtime. See http://cliquet.readthedocs.org/en/latest/reference/configuration.html?highlight=notification#notifications

@Natim

This comment has been minimized.

Contributor

Natim commented Nov 2, 2015

My use case is simple, a user send an archive containing source code. I extract it somewhere in the file system and compile source code (maybe use some kind of container to avoid security flaws) then store the result of the operation in the db. Of course this operations should be asynchronous because my user don't want to wait the compilation time to have a response.

When you say store the result in the DB do you mean in a resource that you can call with the HTTP API?

If it is the case, I would suggest you to just add a cliquet.Service that will handle the file upload and then notify a worker that will build the source code and then use the Kinto.py client to store the results in the database.

@leplatrem

This comment has been minimized.

Contributor

leplatrem commented Nov 2, 2015

I am building a Cliquet-based app. I need to do some server side operations and there is not current way to notify my server of changes in Kinto, right?

From what I understand, you were about to develop a Cliquet-based app because you needed to be notified of the changes?

Since recent versions, we have a notifications system, that every Cliquet-based application can use. Kinto can hence take advantage of it. It is not yet properly documented though (#work-in-progress).

You can either:

  • build a Cliquet extension, that adds a new view to receive uploads and build the async part manually to manipulate the records once compilation was done (as Remy suggested)
  • use a Kinto instance with the Redis events listeners setup, and perform your asynchronous actions elsewhere using the Redis list as a queue of incoming messages. But that would imply submitting your source code as test field inside records, which might not be ideal depending on the size.

A mix of the two is imaginable but it depends on your skills and understanding of the overall ecosystem. We would very happy to help on that, since it would be a great to improve our documentation and pluggabilty :)

Don't hesitate to ask more questions, if our feedback does not feel helpful :)

Enjoy !

@leplatrem leplatrem added the protocol label Nov 26, 2015

@tarekziade

This comment has been minimized.

Contributor

tarekziade commented May 20, 2016

@tarekziade tarekziade closed this May 20, 2016

glasserc pushed a commit that referenced this issue May 20, 2016

Merge pull request #251 from Kinto/236-backend-error-on-implicit-crea…
…tion

Fix 500 if backend fails during implicit creation on default bucket (fixes #236)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment