Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sparsely populated fields in sources can be omitted from the graphql schema #3913

Closed
britt opened this issue Feb 8, 2018 · 12 comments
Closed
Labels
type: question or discussion Issue discussing or asking a question about Gatsby

Comments

@britt
Copy link
Contributor

britt commented Feb 8, 2018

Description

I am loading data from Google Sheets using the gatsby-source-google-sheets plugin. The worksheet has the fields:

  • title
  • dateliked
  • description
  • url
  • notes

The notes field is empty 90+% of the time. It comes from the plugin as a null value. I noticed that it was not appearing in the graphql schema created by the plugin. So, I edited the plugin to supply an empty string instead of a null when calling createNode, and the field appeared in the schema. My guess was that Gatsby was looking at the first value or some sample of values to build the schema.

I am not sure if this is a bug or intended behavior. If it is intended behavior then it would be nice to document it in createNode.

Environment

Gatsby version: 1.9.158
Node.js version: v6.11.4
Operating System: MacOS High Sierra

File contents (if changed):

gatsby-config.js:
package.json:
gatsby-node.js:
gatsby-browser.js:
gatsby-ssr.js:

Actual result

Sparsely populated fields are not included in the graphql schema.

Expected behavior

Field should be included in the graphql schema.

Steps to reproduce

  1. Install and configure the google sheets plugin

  2. Create a Google sheet with a lot of rows, some of which are sparsely populated.

  3. Run gatsby develop

  4. Open the graphiQL editor

  5. query allGoogleSheet<WORKSHEET NAME>Row and attempt to select all fields

...

@fk fk added API/Plugins type: question or discussion Issue discussing or asking a question about Gatsby labels Feb 8, 2018
@m-allanson
Copy link
Contributor

@britt Gatsby creates it's schema from the data you have. As long as one of your notes fields has data in it, that should be enough. If they're all empty then Gatsby won't add a notes field to its schema.

If you have data in at least one notes field, but it's not showing in the schema at all then that sounds like a bug. Would you be able to create an example repo that demonstrates the problem?

You might also be interested in #3344, where there's work happening to allow schemas to be manually defined.

@pieh
Copy link
Contributor

pieh commented Feb 8, 2018

You are right with assumption that we extract sample values - but if something is null we just skip it. My guess there are conflicting types which gatsby will skip during schema creation. Now it skips it silently - I posted PR yesterday to log this so it's easier to identify reason why some fields are not there - #3905

I will check this later, maybe we do have issue in this case. Can you share sample worksheet with this issue so I could easily reproduce it?

@KyleAMathews
Copy link
Contributor

Do we skip nulls though?

@pieh
Copy link
Contributor

pieh commented Feb 8, 2018

In general we do skip them, but this might be some edge case that isn't handled properly. I will look into it and try to recreate described setup and see if I can reproduce this issue.

@britt
Copy link
Contributor Author

britt commented Feb 8, 2018

What I observed was that the field was mostly null, but there were some values in it (>10% out of ~1000 rows) and it got skipped on schema creation.

@pieh
Copy link
Contributor

pieh commented Feb 8, 2018

I tried to reproduce using this sheet - https://docs.google.com/spreadsheets/d/19Q6wT_RKouS9PQTBxsAiek8fv-fSpCzO2zXB1tL_W-Y/edit#gid=0 and can't reproduce it:

84928e8a6791512e48456eed3585f667

I'm almost certain that you have some values that are converted to non-null non-string types by plugin. According to https://www.npmjs.com/package/gatsby-source-google-sheets :

  • Numbers (as determined by isNan()) are converted to numbers
  • "TRUE"/"FALSE" converted to boolean true/false

If you have any of the above it will be treated as conflicting types and skipped in schema.

@britt
Copy link
Contributor Author

britt commented Feb 8, 2018 via email

@pieh
Copy link
Contributor

pieh commented Feb 8, 2018

I definitely am able to reproduce this bug with your data - it is related to note that has just key emoji in it. Will investigate why it happens.

@pieh
Copy link
Contributor

pieh commented Feb 8, 2018

For now you can prepand some text to that note to fix that

@pieh
Copy link
Contributor

pieh commented Feb 8, 2018

Ok, so gatsby-source-google-sheets is what is doing this: https://github.com/brandonmp/gatsby-source-google-sheets/blob/master/src/fetch-sheet.js#L57 it replace any non basic Latin alphabet character from cell value before checking for number (that check will assume that empty string is a number).

I will post issue about it there.

For now just don't use notes that exclusively use emotes and You will be fine.

@britt
Copy link
Contributor Author

britt commented Feb 8, 2018 via email

@pieh
Copy link
Contributor

pieh commented Feb 26, 2018

This was just fixed in source plugin - update your google sheets plugin and enjoy describing stuff with nothing but emoji 🎉 👍

@pieh pieh closed this as completed Feb 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: question or discussion Issue discussing or asking a question about Gatsby
Projects
None yet
Development

No branches or pull requests

5 participants