Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Break upsert() into component parts #35

Open
goatchurchprime opened this issue Oct 3, 2011 · 1 comment
Open

Break upsert() into component parts #35

goatchurchprime opened this issue Oct 3, 2011 · 1 comment

Comments

@goatchurchprime
Copy link

Function is here:
https://github.com/okfn/webstore/blob/master/webstore/views.py#L203

In scraperwiki this is the save(unique_keys=[], data={}) function.

It makes the following distinct processes:

  1. create table if not exists
  2. add columns to existing table [ done with _ensure_columns ]
  3. create an index table for unique_keys [ method from scraperwiki, not webstore ]
  4. update_row (or insert row) of the table

Unique keys are fetched from query_string by request.args.getlist('unique')

The scraperwiki version has an extra jargtypes={key:type} to define the types of the columns and avoid the derivation of variable types happening at the wrong side of the json interface.

Option 1:

Combine steps 1,2,3 into one table schema updating function, and then 4 can be a normal sql update function

Option 2:

Get the client to fetch the schema from webstore at the start and then perform all the computation go generate the schema updates there

@goatchurchprime
Copy link
Author

The current upsert function allows bulk uploading of records from a csv file directly.

Proposal is to keep this function but rewrite it to expose two further functions in the interface:

  1. a schema update function, which is like making the following function writable
    GET /{user-name}/{db-name}/{table-name}/schema

This is not the same as exposing the "CREATE TABLE IF NOT EXISTS" function because that function does not alter tables. It would provide a route towards extra altering functions, such as dropping columns or changing their type -- features which are not given by the underlying database.

This function takes the jargtypes thing as it stands.

  1. An upsert() function taking a list of data rows (filling in missing elements with nulls).

[Discussion to follow of whether unique_keys information is part of the schema, or part of the upsert function.]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant