Sparsely populated fields in sources can be omitted from the graphql schema

[britt]

Description

I am loading data from Google Sheets using the gatsby-source-google-sheets plugin. The worksheet has the fields:

• title
• dateliked
• description
• url
• notes

The notes field is empty 90+% of the time. It comes from the plugin as a null value. I noticed that it was not appearing in the graphql schema created by the plugin. So, I edited the plugin to supply an empty string instead of a null when calling createNode, and the field appeared in the schema. My guess was that Gatsby was looking at the first value or some sample of values to build the schema.

I am not sure if this is a bug or intended behavior. If it is intended behavior then it would be nice to document it in createNode.

Environment

Gatsby version: 1.9.158
Node.js version: v6.11.4
Operating System: MacOS High Sierra

File contents (if changed):

gatsby-config.js:
package.json:
gatsby-node.js:
gatsby-browser.js:
gatsby-ssr.js:

Actual result

Sparsely populated fields are not included in the graphql schema.

Expected behavior

Field should be included in the graphql schema.

Steps to reproduce

1. Install and configure the google sheets plugin

2. Create a Google sheet with a lot of rows, some of which are sparsely populated.

3. Run gatsby develop

4. Open the graphiQL editor

5. query allGoogleSheet<WORKSHEET NAME>Row and attempt to select all fields

...

[m-allanson]

@britt Gatsby creates it's schema from the data you have. As long as one of your notes fields has data in it, that should be enough. If they're all empty then Gatsby won't add a notes field to its schema.

If you have data in at least one notes field, but it's not showing in the schema at all then that sounds like a bug. Would you be able to create an example repo that demonstrates the problem?

You might also be interested in #3344, where there's work happening to allow schemas to be manually defined.

[pieh]

You are right with assumption that we extract sample values - but if something is null we just skip it. My guess there are conflicting types which gatsby will skip during schema creation. Now it skips it silently - I posted PR yesterday to log this so it's easier to identify reason why some fields are not there - #3905

I will check this later, maybe we do have issue in this case. Can you share sample worksheet with this issue so I could easily reproduce it?

[KyleAMathews]

Do we skip nulls though?

[pieh]

In general we do skip them, but this might be some edge case that isn't handled properly. I will look into it and try to recreate described setup and see if I can reproduce this issue.

[britt]

What I observed was that the field was mostly null, but there were some values in it (>10% out of ~1000 rows) and it got skipped on schema creation.

[pieh]

I tried to reproduce using this sheet - https://docs.google.com/spreadsheets/d/19Q6wT_RKouS9PQTBxsAiek8fv-fSpCzO2zXB1tL_W-Y/edit#gid=0 and can't reproduce it:

I'm almost certain that you have some values that are converted to non-null non-string types by plugin. According to https://www.npmjs.com/package/gatsby-source-google-sheets :

• Numbers (as determined by isNan()) are converted to numbers
• "TRUE"/"FALSE" converted to boolean true/false

If you have any of the above it will be treated as conflicting types and skipped in schema.

[britt]

This is the sheet I used that caused problems. Without altering the plugin code to sub empty string for null, the "notes" field simply did not appear. https://docs.google.com/spreadsheets/d/1dFrxOeknJy1nfS85eVhWROR3U9nFF3Zl8qOiOqHxfBY/edit?usp=sharing

…

On Thu, Feb 8, 2018 at 11:37 AM Michal Piechowiak ***@***.***> wrote: I tried to reproduce using this sheet - https://docs.google.com/spreadsheets/d/19Q6wT_RKouS9PQTBxsAiek8fv-fSpCzO2zXB1tL_W-Y/edit#gid=0 and can't reproduce it: [image: 84928e8a6791512e48456eed3585f667] <https://user-images.githubusercontent.com/419821/35993681-65ff0cf2-0d0e-11e8-98e4-07bb51e7ad1c.png> I'm almost certain that you have some values that are converted to non-null non-string types by plugin. According to https://www.npmjs.com/package/gatsby-source-google-sheets : - Numbers (as determined by isNan()) are converted to numbers - "TRUE"/"FALSE" converted to boolean true/false If you have any of the above it will be treated as conflicting types and skipped in schema. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3913 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAE69JacKWzpooOkwx_tOiGTiRNVVwNks5tS0uBgaJpZM4R91o3> .

[pieh]

I definitely am able to reproduce this bug with your data - it is related to note that has just key emoji in it. Will investigate why it happens.

[pieh]

For now you can prepand some text to that note to fix that

[pieh]

Ok, so gatsby-source-google-sheets is what is doing this: https://github.com/brandonmp/gatsby-source-google-sheets/blob/master/src/fetch-sheet.js#L57 it replace any non basic Latin alphabet character from cell value before checking for number (that check will assume that empty string is a number).

I will post issue about it there.

For now just don't use notes that exclusively use emotes and You will be fine.

[britt]

Thanks so much for figuring that out. I tried to debug the plugin but missed that apparently. I guess I was intercepting it before it pruned those values. Sorry for the goose chase.

…

On Thu, Feb 8, 2018 at 2:19 PM Michal Piechowiak ***@***.***> wrote: Ok, so gatsby-source-google-sheets is what is doing this: https://github.com/brandonmp/gatsby-source-google-sheets/blob/master/src/fetch-sheet.js#L57 it replace any non basic Latin alphabet character from cell value before checking for number (that check will assume that empty string is a number). I will post issue about it there. For now just don't use notes that exclusively use emotes and You will be fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3913 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAE6xh3qxSgxAtL0Z2NEcHaNAXuOwQPks5tS3L8gaJpZM4R91o3> .

[pieh]

This was just fixed in source plugin - update your google sheets plugin and enjoy describing stuff with nothing but emoji 🎉 👍