Support non UTF-8 Encoding for Sources

[dluftspring]

For both file and database sources it would be great if you could support non utf-8 encoding as a configuration option

[flarco]

Not sure if this is feasible... What types of non-utf8 encodings are you thinking of?

[dluftspring]

latin-1 is the one I encounter most but I understand if this isn't worth pursuing or isn't possible. At a certain point you can just say data in the databases should be utf-8 and there should be processes to make sure weird characters don't get in there

[flarco]

Hey, I pre-released a new version: https://github.com/slingdata-io/sling-cli/releases/tag/v1.0.31
Can you download the binary, test and let me know if it works? You can use the transforms: [ decode_latin1 ] option like this:

source: SNOWFLAKE_DEV
target: DOCKER_POSTGRES

defaults:
  mode: truncate
  source_options:
    transforms: [ decode_latin1 ]

streams:
  snowflake_schema.snowflake_table:
    object: public.my_postgres_table

Here are the options: https://github.com/slingdata-io/sling-cli/blob/main/core/sling/tranforms.go#L20-L27

[dluftspring]

Testing with the pre-release this looks like it's working properly to me!