Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sparsely populated fields in sources can be omitted from the graphql schema #3913

Closed
britt opened this issue Feb 8, 2018 · 12 comments

Comments

Projects
None yet
5 participants
@britt
Copy link
Contributor

commented Feb 8, 2018

Description

I am loading data from Google Sheets using the gatsby-source-google-sheets plugin. The worksheet has the fields:

  • title
  • dateliked
  • description
  • url
  • notes

The notes field is empty 90+% of the time. It comes from the plugin as a null value. I noticed that it was not appearing in the graphql schema created by the plugin. So, I edited the plugin to supply an empty string instead of a null when calling createNode, and the field appeared in the schema. My guess was that Gatsby was looking at the first value or some sample of values to build the schema.

I am not sure if this is a bug or intended behavior. If it is intended behavior then it would be nice to document it in createNode.

Environment

Gatsby version: 1.9.158
Node.js version: v6.11.4
Operating System: MacOS High Sierra

File contents (if changed):

gatsby-config.js:
package.json:
gatsby-node.js:
gatsby-browser.js:
gatsby-ssr.js:

Actual result

Sparsely populated fields are not included in the graphql schema.

Expected behavior

Field should be included in the graphql schema.

Steps to reproduce

  1. Install and configure the google sheets plugin

  2. Create a Google sheet with a lot of rows, some of which are sparsely populated.

  3. Run gatsby develop

  4. Open the graphiQL editor

  5. query allGoogleSheet<WORKSHEET NAME>Row and attempt to select all fields

...

@m-allanson

This comment has been minimized.

Copy link
Member

commented Feb 8, 2018

@britt Gatsby creates it's schema from the data you have. As long as one of your notes fields has data in it, that should be enough. If they're all empty then Gatsby won't add a notes field to its schema.

If you have data in at least one notes field, but it's not showing in the schema at all then that sounds like a bug. Would you be able to create an example repo that demonstrates the problem?

You might also be interested in #3344, where there's work happening to allow schemas to be manually defined.

@pieh

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

You are right with assumption that we extract sample values - but if something is null we just skip it. My guess there are conflicting types which gatsby will skip during schema creation. Now it skips it silently - I posted PR yesterday to log this so it's easier to identify reason why some fields are not there - #3905

I will check this later, maybe we do have issue in this case. Can you share sample worksheet with this issue so I could easily reproduce it?

@KyleAMathews

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

Do we skip nulls though?

@pieh

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

In general we do skip them, but this might be some edge case that isn't handled properly. I will look into it and try to recreate described setup and see if I can reproduce this issue.

@britt

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2018

What I observed was that the field was mostly null, but there were some values in it (>10% out of ~1000 rows) and it got skipped on schema creation.

@pieh

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

I tried to reproduce using this sheet - https://docs.google.com/spreadsheets/d/19Q6wT_RKouS9PQTBxsAiek8fv-fSpCzO2zXB1tL_W-Y/edit#gid=0 and can't reproduce it:

84928e8a6791512e48456eed3585f667

I'm almost certain that you have some values that are converted to non-null non-string types by plugin. According to https://www.npmjs.com/package/gatsby-source-google-sheets :

  • Numbers (as determined by isNan()) are converted to numbers
  • "TRUE"/"FALSE" converted to boolean true/false

If you have any of the above it will be treated as conflicting types and skipped in schema.

@britt

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2018

@pieh

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

I definitely am able to reproduce this bug with your data - it is related to note that has just key emoji in it. Will investigate why it happens.

@pieh

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

For now you can prepand some text to that note to fix that

@pieh

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

Ok, so gatsby-source-google-sheets is what is doing this: https://github.com/brandonmp/gatsby-source-google-sheets/blob/master/src/fetch-sheet.js#L57 it replace any non basic Latin alphabet character from cell value before checking for number (that check will assume that empty string is a number).

I will post issue about it there.

For now just don't use notes that exclusively use emotes and You will be fine.

@britt

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2018

@pieh

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2018

This was just fixed in source plugin - update your google sheets plugin and enjoy describing stuff with nothing but emoji 🎉 👍

@pieh pieh closed this Feb 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.