Pass custom encoding to loadNodeContent method

[alexandrtovmach]

Summary

Pass custom encoding as argument to loadNodeContent method. Currently, it's hardcoded as utf-8, and there is no way to load different content.

Basic example

I'm using loadNodeContent to get content from files:

Files used windows-1251 encoding:

After loading with loadNodeContent symbols become broken:

Motivation

I faced with project that used windows-1251 encoding in files, and I believe that's helpful for other Japanese/Chinese symbol encodings. Also it's useful for transformer plugins development.

I'll prepare PR with this feature. There is just few simple changes in gatsby-source-filesystem and gatsby core

P.S. This project from 2000s, so sorry for deprecated HTML markup on screenshots 🤷‍♂️

[vladar]

Thanks for opening this, @alexandrtovmach !

This sounds like a valid problem. But I am not sure if adding encoding as an argument to loadNodeContent is the best way to go here. loadNodeContent is basically an interface that encapsulates knowledge on how to load any node (including binary streams).

So I think the information about an encoding should be somehow embedded in the node itself. Probably the same way we do with mimeTypes. Then the plugin that is responsible for actually loading the node could properly convert it to utf-8.

I realize that the scope for this change would be much bigger and not 100% sure I am right here, so let's wait for others input, especially @pieh

[pieh]

So I think the information about an encoding should be somehow embedded in the node itself. Probably the same way we do with mimeTypes. Then the plugin that is responsible for actually loading the node could properly convert it to utf-8.

I'm not sure if this would work for filesystem to set encoding on its own, it would need to be hinted by users (or plugins calling loadNodeContent) in some way what encoding to use. Also this would require breaking change as right now we return decoded string and not pre-decoding buffer.

Given above, I would be ok with optional 2nd param (as done in #26174) to force non-utf-8 decoding controlled by transformer plugin (I think most transformers that would use it would be pretty custom, because of how unwieldy it would be to make this user configurable via plugin options and use cases that needs this - most of data is utf-8 these days)