[Track] bugs in csv input format parser

[youngsofun]

Summary

[youngsofun]

@ZhiHanZ a, {'b": "c"}, d is not a valid CSV, to my understanding. can you provide more context for it?

[ZhiHanZ]

@ZhiHanZ a, {'b": "c"}, d is not a valid CSV, to my understanding. can you provide more context for it?

That is some psedo examples, here is some temp file generated from airbyte which could be helpful

ee9e9984-083a-439b-8a1d-f9178181fb01,"{""id"":1,""topic"":""install"",""timestamp"":""2017-08-22T17:22:15Z"",""model"":""install"",""details"":""{\""name\"":\""Conferência Faturamento - Custo - Taxas - Margem - Resumo ano inicial até -2\"",\""description\"":null}""}",2021-07-08 00:58:59.889

[sundy-li]

Need modify

 fn de_text_csv<R: BufferRead>(
        &mut self,
        reader: &mut R,
        _format: &FormatSettings,
    ) -> Result<()> {
        self.buffer.clear();
        reader.read_quoted_text(&mut self.buffer, b'"')?;

        let val = serde_json::from_slice(self.buffer.as_slice())?;
        self.builder.append_value(val);
        Ok(())
    }

If the first char is '{', we should consume to the next matched '}';

So do '[' .

[youngsofun]

I think csv should do know nothing about { or [ ?
do we support " as default escape char (instead of \) or need to specify it before parse?

embedded json field in csv/tsv should be serialized to a string, and then escaped with the csv/tsv rule

[wubx]

track: #5586

[wubx]

@ZhiHanZ a, {'b": "c"}, d is not a valid CSV, to my understanding. can you provide more context for it?

That is some psedo examples, here is some temp file generated from airbyte which could be helpful

ee9e9984-083a-439b-8a1d-f9178181fb01,"{""id"":1,""topic"":""install"",""timestamp"":""2017-08-22T17:22:15Z"",""model"":""install"",""details"":""{\""name\"":\""Conferência Faturamento - Custo - Taxas - Margem - Resumo ano inicial até -2\"",\""description\"":null}""}",2021-07-08 00:58:59.889

if support this feature, github archive file is very easy to load to databend.

[youngsofun]

@wubx @ZhiHanZ this case is fixed by #6524 (comment)

@ZhiHanZ a, {'b": "c"}, d is not a valid CSV, to my understanding. can you provide more context for it?

That is some psedo examples, here is some temp file generated from airbyte which could be helpful

ee9e9984-083a-439b-8a1d-f9178181fb01,"{""id"":1,""topic"":""install"",""timestamp"":""2017-08-22T17:22:15Z"",""model"":""install"",""details"":""{\""name\"":\""Conferência Faturamento - Custo - Taxas - Margem - Resumo ano inicial até -2\"",\""description\"":null}""}",2021-07-08 00:58:59.889

if support this feature, github archive file is very easy to load to databend.