Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google Spreadsheet Import #24

Closed
rufuspollock opened this issue Apr 5, 2013 · 16 comments
Closed

Google Spreadsheet Import #24

rufuspollock opened this issue Apr 5, 2013 · 16 comments

Comments

@rufuspollock
Copy link
Contributor

@rufuspollock rufuspollock commented Apr 5, 2013

@rufuspollock rufuspollock mentioned this issue Apr 8, 2013
8 of 10 tasks complete
@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented Apr 9, 2013

Functioning google app script library plus instructions

Demo spreadsheet

  • Make a copy (unfortunately you cannot run app scripts if you are not owner!)
  • See data packages menu!
@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented May 7, 2013

I'm going to mark as fixed on basis that publishing will take some time (and we probably want to wait to give this some testing).

@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented Jan 23, 2014

Re-opening as this needs more work and data package spec has updated a little since then.

@rufuspollock rufuspollock reopened this Jan 23, 2014
@mattfullerton
Copy link

@mattfullerton mattfullerton commented Mar 15, 2014

A few points:

  • The demo spreadsheet appeared to be referring to version 3 of the library; I changed it to 5
  • We should add support for multiple CSV files
  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n
  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now
  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

If I were to turn this some of this stuff into issues (and I'm not sure its necessary), where should I put it - new repo, data-protocols or here? As for an exporter, I guess it would merit its own repo.

@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented Mar 16, 2014

@mattfullerton great points. Here are some thoughts:

  • The demo spreadsheet appeared to be referring to version 3 of the library; I changed it to 5

Great

  • We should add support for multiple CSV files

Sure though let's start with the simplest thing :-)

  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n

I feel it could be worth us creating a generic Tabular Data Package API that takes care of doing all the relevant CSV option parsing etc and providing nice standard JSON. This would greatly simplify writing apps like this and avoid repeating the same code in many different locations. In some sense we already have some of this in data.okfn.org webapp (could improve this) - see "Data packages somewhere online" section of #19

  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now

I think its time we booted its own repo - as lead on this do you want to create it :-) I suggest naming as: datapackage-to-google-spreadsheet (from can be separate).

  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

Exporter idea makes sense though I think something activated within the google spreadsheet is nice (even if it just calls the nodejs app) - i think for non-expert users this will be simpler (but we can start with the app)

If I were to turn this some of this stuff into issues (and I'm not sure its necessary), where should I put it - new repo, data-protocols or here? As for an exporter, I guess it would merit its own repo.

As above I think we want its own repo for importer (we could also create one for exporter when time comes).

@mattfullerton
Copy link

@mattfullerton mattfullerton commented Mar 19, 2014

  • We should add support for multiple CSV files

Sure though let's start with the simplest thing :-)

  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n

I feel it could be worth us creating a generic Tabular Data Package API that takes care of doing all the relevant CSV option parsing etc and providing nice standard JSON. This would greatly simplify writing apps like this and avoid repeating the same code in many different locations. In some sense we already have some of this in data.okfn.org webapp (could improve this) - see "Data packages somewhere online" section of #19

So the key challenge is to ensure that this API is available to any github-hosted data package(?). I'll concentrate on normalised CSV for now then so we can move over to the API later.

  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now

I think its time we booted its own repo - as lead on this do you want to create it :-) I suggest naming as: datapackage-to-google-spreadsheet (from can be separate).

Done: https://github.com/okfn/datapackage-to-google-spreadsheet

I'll add the code, some documentation and issues later.

  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

Exporter idea makes sense though I think something activated within the google spreadsheet is nice (even if it just calls the nodejs app) - i think for non-expert users this will be simpler (but we can start with the app)

I agree completely. Also a central API could take over some of the workload of putting the package together, the only GS-specific bit would be pulling out the meta data. Its probably good to wait a little until the exporter/datastore pusher firms up and then we'll know what we're connecting to.

@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented Mar 23, 2014

All looks good - I imagine first step is commit existing script into https://github.com/okfn/datapackage-to-google-spreadsheet and then go from there.

@mattfullerton
Copy link

@mattfullerton mattfullerton commented Mar 24, 2014

Yes, can you give me write access to the repository? I transferred it to OKFN without noticing that I don't have write access to all OKFN repos.

@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented Mar 24, 2014

@mattfullerton
Copy link

@mattfullerton mattfullerton commented Mar 26, 2014

Thanks, everything's been added and documented so we're off to a good start

@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented Jun 7, 2014

@mattfullerton how is this going? Do you want to update the tools page?

@mattfullerton
Copy link

@mattfullerton mattfullerton commented Jul 4, 2014

Finally getting back to you on this. I was on a hiatus for several months due to a new child and a new job. Getting back to it now and have tried to prepare an extensive CSV test data set based on http://dataprotocols.org/json-table-schema/#field-types: https://docs.google.com/spreadsheets/d/1mSHxHi8u176SYPmqGK5RshOgD-nAWvjjQ9REJ_Z43Ks/edit?usp=sharing

The idea is to only support one format per type in line with your API proposal. Let me know what you think. I still have to prepare the JSON schema for it and then start testing and extending the importer accordingly.

@danfowler
Copy link
Contributor

@danfowler danfowler commented Sep 30, 2016

@danfowler danfowler closed this Sep 30, 2016
@rufuspollock
Copy link
Contributor Author

@rufuspollock rufuspollock commented Sep 30, 2016

@danfowler the point of this repo was to track high level progress though - to understand the big tasks and when they were complete. Is that now tracked somewhere else?

@danfowler
Copy link
Contributor

@danfowler danfowler commented Sep 30, 2016

@rgrp trying to get to the point where this repo tracks high level progress for active, current work. This particular issue hasn't been touched in more than two years and only refers to one project. Now that https://github.com/frictionlessdata/datapackage-to-google-spreadsheet has been created and that there is no work currently being done on it, I believe that that repo is the natural home for this issue until we can agree it can be closed. We can start a new issue on this repo to track high level progress for the next set of work on the Google Sheets integration when it happens.

For other "high level" ideas about the project, with or without their own GitHub repos, we have a User Stories Trello board: https://trello.com/b/MGC4RpTZ/frictionless-data-user-stories

#262

@mattfullerton
Copy link

@mattfullerton mattfullerton commented Oct 1, 2016

@danfowler @rgrp Sounds good to me. I haven't been able to follow frictionlessdata progress recently apart from noticing that an awful lot is happening. In that sense I have no idea any more of what is still needed here and where it fits in. Happy to help though if there is something to progress here. Or rather not here but in the other repo 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants