Google Spreadsheet Import #24

Closed
rufuspollock opened this Issue Apr 5, 2013 · 16 comments

Comments

Projects
None yet
4 participants
@rufuspollock
Contributor

rufuspollock commented Apr 5, 2013

@rufuspollock rufuspollock referenced this issue Apr 8, 2013

Closed

For Launch #20

8 of 10 tasks complete
@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock Apr 9, 2013

Contributor

Functioning google app script library plus instructions

Demo spreadsheet

  • Make a copy (unfortunately you cannot run app scripts if you are not owner!)
  • See data packages menu!
Contributor

rufuspollock commented Apr 9, 2013

Functioning google app script library plus instructions

Demo spreadsheet

  • Make a copy (unfortunately you cannot run app scripts if you are not owner!)
  • See data packages menu!
@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock May 7, 2013

Contributor

I'm going to mark as fixed on basis that publishing will take some time (and we probably want to wait to give this some testing).

Contributor

rufuspollock commented May 7, 2013

I'm going to mark as fixed on basis that publishing will take some time (and we probably want to wait to give this some testing).

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock Jan 23, 2014

Contributor

Re-opening as this needs more work and data package spec has updated a little since then.

Contributor

rufuspollock commented Jan 23, 2014

Re-opening as this needs more work and data package spec has updated a little since then.

@rufuspollock rufuspollock reopened this Jan 23, 2014

@mattfullerton

This comment has been minimized.

Show comment
Hide comment
@mattfullerton

mattfullerton Mar 15, 2014

A few points:

  • The demo spreadsheet appeared to be referring to version 3 of the library; I changed it to 5
  • We should add support for multiple CSV files
  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n
  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now
  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

If I were to turn this some of this stuff into issues (and I'm not sure its necessary), where should I put it - new repo, data-protocols or here? As for an exporter, I guess it would merit its own repo.

A few points:

  • The demo spreadsheet appeared to be referring to version 3 of the library; I changed it to 5
  • We should add support for multiple CSV files
  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n
  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now
  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

If I were to turn this some of this stuff into issues (and I'm not sure its necessary), where should I put it - new repo, data-protocols or here? As for an exporter, I guess it would merit its own repo.

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock Mar 16, 2014

Contributor

@mattfullerton great points. Here are some thoughts:

  • The demo spreadsheet appeared to be referring to version 3 of the library; I changed it to 5

Great

  • We should add support for multiple CSV files

Sure though let's start with the simplest thing :-)

  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n

I feel it could be worth us creating a generic Tabular Data Package API that takes care of doing all the relevant CSV option parsing etc and providing nice standard JSON. This would greatly simplify writing apps like this and avoid repeating the same code in many different locations. In some sense we already have some of this in data.okfn.org webapp (could improve this) - see "Data packages somewhere online" section of #19

  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now

I think its time we booted its own repo - as lead on this do you want to create it :-) I suggest naming as: datapackage-to-google-spreadsheet (from can be separate).

  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

Exporter idea makes sense though I think something activated within the google spreadsheet is nice (even if it just calls the nodejs app) - i think for non-expert users this will be simpler (but we can start with the app)

If I were to turn this some of this stuff into issues (and I'm not sure its necessary), where should I put it - new repo, data-protocols or here? As for an exporter, I guess it would merit its own repo.

As above I think we want its own repo for importer (we could also create one for exporter when time comes).

Contributor

rufuspollock commented Mar 16, 2014

@mattfullerton great points. Here are some thoughts:

  • The demo spreadsheet appeared to be referring to version 3 of the library; I changed it to 5

Great

  • We should add support for multiple CSV files

Sure though let's start with the simplest thing :-)

  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n

I feel it could be worth us creating a generic Tabular Data Package API that takes care of doing all the relevant CSV option parsing etc and providing nice standard JSON. This would greatly simplify writing apps like this and avoid repeating the same code in many different locations. In some sense we already have some of this in data.okfn.org webapp (could improve this) - see "Data packages somewhere online" section of #19

  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now

I think its time we booted its own repo - as lead on this do you want to create it :-) I suggest naming as: datapackage-to-google-spreadsheet (from can be separate).

  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

Exporter idea makes sense though I think something activated within the google spreadsheet is nice (even if it just calls the nodejs app) - i think for non-expert users this will be simpler (but we can start with the app)

If I were to turn this some of this stuff into issues (and I'm not sure its necessary), where should I put it - new repo, data-protocols or here? As for an exporter, I guess it would merit its own repo.

As above I think we want its own repo for importer (we could also create one for exporter when time comes).

@mattfullerton

This comment has been minimized.

Show comment
Hide comment
@mattfullerton

mattfullerton Mar 19, 2014

  • We should add support for multiple CSV files

Sure though let's start with the simplest thing :-)

  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n

I feel it could be worth us creating a generic Tabular Data Package API that takes care of doing all the relevant CSV option parsing etc and providing nice standard JSON. This would greatly simplify writing apps like this and avoid repeating the same code in many different locations. In some sense we already have some of this in data.okfn.org webapp (could improve this) - see "Data packages somewhere online" section of #19

So the key challenge is to ensure that this API is available to any github-hosted data package(?). I'll concentrate on normalised CSV for now then so we can move over to the API later.

  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now

I think its time we booted its own repo - as lead on this do you want to create it :-) I suggest naming as: datapackage-to-google-spreadsheet (from can be separate).

Done: https://github.com/okfn/datapackage-to-google-spreadsheet

I'll add the code, some documentation and issues later.

  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

Exporter idea makes sense though I think something activated within the google spreadsheet is nice (even if it just calls the nodejs app) - i think for non-expert users this will be simpler (but we can start with the app)

I agree completely. Also a central API could take over some of the workload of putting the package together, the only GS-specific bit would be pulling out the meta data. Its probably good to wait a little until the exporter/datastore pusher firms up and then we'll know what we're connecting to.

  • We should add support for multiple CSV files

Sure though let's start with the simplest thing :-)

  • We should add support for csv dialects, some of the parameters would just have to be passed through to the CSV reading code
  • According to the SDF spec, \r\n is just as acceptable as \n, so it should be checked for in addition to \n

I feel it could be worth us creating a generic Tabular Data Package API that takes care of doing all the relevant CSV option parsing etc and providing nice standard JSON. This would greatly simplify writing apps like this and avoid repeating the same code in many different locations. In some sense we already have some of this in data.okfn.org webapp (could improve this) - see "Data packages somewhere online" section of #19

So the key challenge is to ensure that this API is available to any github-hosted data package(?). I'll concentrate on normalised CSV for now then so we can move over to the API later.

  • Where is the best place to put some minimal documentation, i.e. the link to the demo spreadsheet for now

I think its time we booted its own repo - as lead on this do you want to create it :-) I suggest naming as: datapackage-to-google-spreadsheet (from can be separate).

Done: https://github.com/okfn/datapackage-to-google-spreadsheet

I'll add the code, some documentation and issues later.

  • For an exporter, I was thinking of writing a very small node.js webapp that would export the datapackage.json file using the metadata from the Google Spreadsheets API. Thoughts?

Exporter idea makes sense though I think something activated within the google spreadsheet is nice (even if it just calls the nodejs app) - i think for non-expert users this will be simpler (but we can start with the app)

I agree completely. Also a central API could take over some of the workload of putting the package together, the only GS-specific bit would be pulling out the meta data. Its probably good to wait a little until the exporter/datastore pusher firms up and then we'll know what we're connecting to.

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock Mar 23, 2014

Contributor

All looks good - I imagine first step is commit existing script into https://github.com/okfn/datapackage-to-google-spreadsheet and then go from there.

Contributor

rufuspollock commented Mar 23, 2014

All looks good - I imagine first step is commit existing script into https://github.com/okfn/datapackage-to-google-spreadsheet and then go from there.

@mattfullerton

This comment has been minimized.

Show comment
Hide comment
@mattfullerton

mattfullerton Mar 24, 2014

Yes, can you give me write access to the repository? I transferred it to OKFN without noticing that I don't have write access to all OKFN repos.

Yes, can you give me write access to the repository? I transferred it to OKFN without noticing that I don't have write access to all OKFN repos.

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
Contributor

rufuspollock commented Mar 24, 2014

@mattfullerton

This comment has been minimized.

Show comment
Hide comment
@mattfullerton

mattfullerton Mar 26, 2014

Thanks, everything's been added and documented so we're off to a good start

Thanks, everything's been added and documented so we're off to a good start

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock Jun 7, 2014

Contributor

@mattfullerton how is this going? Do you want to update the tools page?

Contributor

rufuspollock commented Jun 7, 2014

@mattfullerton how is this going? Do you want to update the tools page?

@mattfullerton

This comment has been minimized.

Show comment
Hide comment
@mattfullerton

mattfullerton Jul 4, 2014

Finally getting back to you on this. I was on a hiatus for several months due to a new child and a new job. Getting back to it now and have tried to prepare an extensive CSV test data set based on http://dataprotocols.org/json-table-schema/#field-types: https://docs.google.com/spreadsheets/d/1mSHxHi8u176SYPmqGK5RshOgD-nAWvjjQ9REJ_Z43Ks/edit?usp=sharing

The idea is to only support one format per type in line with your API proposal. Let me know what you think. I still have to prepare the JSON schema for it and then start testing and extending the importer accordingly.

Finally getting back to you on this. I was on a hiatus for several months due to a new child and a new job. Getting back to it now and have tried to prepare an extensive CSV test data set based on http://dataprotocols.org/json-table-schema/#field-types: https://docs.google.com/spreadsheets/d/1mSHxHi8u176SYPmqGK5RshOgD-nAWvjjQ9REJ_Z43Ks/edit?usp=sharing

The idea is to only support one format per type in line with your API proposal. Let me know what you think. I still have to prepare the JSON schema for it and then start testing and extending the importer accordingly.

@mattfullerton mattfullerton referenced this issue in oki-archive/datapackage-to-google-spreadsheet Dec 10, 2014

Open

Data package export from Google Spreadsheet? #7

@roll roll added the Status: Backlog label May 6, 2016

@roll roll added backlog and removed Topic: Tools labels Aug 8, 2016

@roll roll modified the milestone: For Launch Aug 11, 2016

@roll roll removed the backlog label Aug 11, 2016

@roll roll removed this from the For Launch milestone Aug 19, 2016

@roll roll added the user-story label Sep 5, 2016

@danfowler danfowler referenced this issue in oki-archive/datapackage-to-google-spreadsheet Sep 30, 2016

Open

Google Spreadsheet Import #8

2 of 3 tasks complete
@danfowler

This comment has been minimized.

Show comment
Hide comment
Contributor

danfowler commented Sep 30, 2016

@danfowler danfowler closed this Sep 30, 2016

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock Sep 30, 2016

Contributor

@danfowler the point of this repo was to track high level progress though - to understand the big tasks and when they were complete. Is that now tracked somewhere else?

Contributor

rufuspollock commented Sep 30, 2016

@danfowler the point of this repo was to track high level progress though - to understand the big tasks and when they were complete. Is that now tracked somewhere else?

@danfowler danfowler removed the user-story label Sep 30, 2016

@danfowler

This comment has been minimized.

Show comment
Hide comment
@danfowler

danfowler Sep 30, 2016

Contributor

@rgrp trying to get to the point where this repo tracks high level progress for active, current work. This particular issue hasn't been touched in more than two years and only refers to one project. Now that https://github.com/frictionlessdata/datapackage-to-google-spreadsheet has been created and that there is no work currently being done on it, I believe that that repo is the natural home for this issue until we can agree it can be closed. We can start a new issue on this repo to track high level progress for the next set of work on the Google Sheets integration when it happens.

For other "high level" ideas about the project, with or without their own GitHub repos, we have a User Stories Trello board: https://trello.com/b/MGC4RpTZ/frictionless-data-user-stories

#262

Contributor

danfowler commented Sep 30, 2016

@rgrp trying to get to the point where this repo tracks high level progress for active, current work. This particular issue hasn't been touched in more than two years and only refers to one project. Now that https://github.com/frictionlessdata/datapackage-to-google-spreadsheet has been created and that there is no work currently being done on it, I believe that that repo is the natural home for this issue until we can agree it can be closed. We can start a new issue on this repo to track high level progress for the next set of work on the Google Sheets integration when it happens.

For other "high level" ideas about the project, with or without their own GitHub repos, we have a User Stories Trello board: https://trello.com/b/MGC4RpTZ/frictionless-data-user-stories

#262

@mattfullerton

This comment has been minimized.

Show comment
Hide comment
@mattfullerton

mattfullerton Oct 1, 2016

@danfowler @rgrp Sounds good to me. I haven't been able to follow frictionlessdata progress recently apart from noticing that an awful lot is happening. In that sense I have no idea any more of what is still needed here and where it fits in. Happy to help though if there is something to progress here. Or rather not here but in the other repo 😉

@danfowler @rgrp Sounds good to me. I haven't been able to follow frictionlessdata progress recently apart from noticing that an awful lot is happening. In that sense I have no idea any more of what is still needed here and where it fits in. Happy to help though if there is something to progress here. Or rather not here but in the other repo 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment