adding support for TEXT fields, remove non-standard ascii characters

[odedelharar]

Added a very basic TEXT field support, while removing non-ascii characters from such a field.
Updated the date and datetime fields to support string representations of unix time.

[cognitom]

@odedelharar thank you for your PR! LGTM about TEXT type. I like it 👍

I assumed that DATE and DATETIME type are string like 2016-11-07 basically. Could you change the code to pass such values?

The code could be like this:

convert: val => {
  if (!val) return 'NULL'
  if (typeof val == 'number') val = moment(val).format('YYYY-MM-DD')
  if (typeof val == 'string') {
    if (/\d+/.test(val)) val = moment(parseInt(val)).format('YYYY-MM-DD')
    if (!/\d{4}-\d{2}-\d{2}/.test(val)) return 'NULL'
  }
  return `"${ val }"`
}

[cognitom]

Could you pls add documentation about TEXT to here, too? Thanks.

[odedelharar]

Updated, now can allow: date format string (YYYY-MM-DD), unix time (number), unix time (string). Can make more options viable, maybe in a future change

[cognitom]

Great! Thank you.

[cognitom]

Oops, I'll remove this line, later.
We need to care about Unicode chars equally.

[cognitom]

I'm fixing the code like this.

.replace(/[\x00-\x1F\x7F]/g, '')

[cognitom]

Umm..., we should keep \x09 and \x10.

.replace(/[\x00-\x08\x11-\x1F\x7F]/g, '')

@odedelharar could you explain the use case you thought?

[odedelharar]

Yes, I had a problem with a field with a non-standard character. MySQL wouldn't allow inserting this line. If we want to keep this line (we should have some safeguard there), we should include all characters allowed in standard DB encoding (unless you have examples of non-standard encoding). So my code allows all common ASCII characters (0-127).

[cognitom]

@odedelharar thanks. I've checked the spec of MySQL: \0 \b \t \n \r \x1a supposed to be allowed. Actually sqlstring module care about those control chars.
https://github.com/mysqljs/sqlstring/blob/master/lib/SqlString.js#L2

At this point, I'd like to remove control chars except above, and escape them by sqlstring.

The meaning of Non-standard depends on what charset we use, I think.
If you need another custom filter, feel free to send a new PR 😉

[odedelharar]

OK, I'll find a way to remove the offending characters without removing anything else. In my case the character was \xEF\xBF\xBD or �.

[odedelharar]

Maybe add an expression like: /[\x00-\x1F\x7F\uFFFD\xEF\xBF\xBD]/g so it also covers the cases I saw

[cognitom]

@odedelharar unfortunately, it seems not a good idea. These chars like \xBF \xBD are actually valid letters in some languages.

[odedelharar]

OK, so maybe we can use /[\x00-\x1F\x7F\uFFFD]/g since \uFFFD (�) is not OK anyway.