Implement column-type-aware CSV import: defer type conversion to INSERT based on table definitions #2403

Copilot · 2025-12-22T02:22:35Z

Summary

Applied maintainer feedback to further optimize the type converter utility.

Changes Made

Type Converter Optimizations (src/15utility.js)

Simplified int regex (Line 1355)
- Before: /^int|int$|^smallint$|^bigint$|^tinyint$/i
- After: /int$/i
- All int types (smallint, bigint, tinyint) are already covered by int$ pattern
Removed trim() call (Line 1376)
- Before: var trimmed = value.trim(); if (trimmed == +trimmed)
- After: if (value == +value)
- Direct comparison without unnecessary trim operation
Boolean conversion using regex (Lines 1396-1397)
- Before: var lower = value.toLowerCase(); return lower === 'true' || lower === '1' || lower === 'yes';
- After: return /^(true|1|yes)$/i.test(value);
- More efficient single regex test instead of toLowerCase + multiple comparisons

Testing Results

✅ All 17 CSV string type tests pass
✅ All 47 CSV-related tests pass
✅ No regressions detected
✅ Code is cleaner and more efficient

Original prompt

This section details on the original issue you should resolve

<issue_title>Wrong datatype after reading FROM CSV </issue_title>
<issue_description>I am very enthusiastic about alasql and it was already a great help to me. Thanks a lot for the great work.
But...

Datatype switches from string to number after FROM CSV import.

CSV sample:
"117.20";"some name"
"88.33";"other name"

SQL:
CREATE TABLE test (id string,name string);
SELECT * INTO test FROM CSV('some csv file');
SELECT * FROM test;

results in:

id name

117.2 some name

88.33 other name

The column 'ID' datatype changed to number and the value is handled as float and trailing Null is cut off.
Bad if this is an identifier.

PS: it seems to work with FROM XLSX, when column is formated as TEXT in Excel
</issue_description>

Comments on the Issue (you are @copilot in this section)
@mathiasrw Hi @casavipiestrick - thank you for the kind words.
Good point. Nice little bug there :)</comment_new>
<comment_new>@mathiasrw
The conversion of numberstrings into numbers has been part of Alasql too long to change. I am happy to let a config flag change how it works.
@mathiasrw Any PR is welcome to fix this - but the fix must be provide in a way so you need to set something like alasql.config.voidstr2intCSV (preferably a better name) to have that new functionality.

Fixes Wrong datatype after reading FROM CSV #1181

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

src/84from.js

test/test-csv-string-type.js

Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

mathiasrw · 2025-12-22T09:15:36Z

I recommend we keep csv data as string until we are inserting them.

If there are col definitions, then we adhire to that (also with things like dates, int, string, float, varchar)

If there are no definitions we behave like now when we do auto convert if we detect numbers (for example into tables provideda s reference data)

mathiasrw · 2025-12-22T09:51:35Z

@copilot

…at INSERT based on table definitions Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

Copilot · 2025-12-22T10:47:51Z

I recommend we keep csv data as string until we are inserting them.

If there are col definitions, then we adhire to that (also with things like dates, int, string, float, varchar)
...

Implemented as requested. CSV data now stays as strings until INSERT. Type conversion happens based on column definitions (STRING, INT, FLOAT, etc.) when available. Without definitions, the csvStringToNumber option controls auto-conversion for backward compatibility. Commit: 588303a

src/15utility.js

src/40select.js

test/test-csv-string-type.js

…ECT INTO, add comprehensive edge case tests Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

test/test-csv-string-type.js

… complete object validation Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

mathiasrw · 2025-12-22T22:49:04Z

src/15utility.js

+ */
+var reTypeConverter = {
+	str: /string|char$|text/i,
+	int: /^int|int$|^smallint$|^bigint$|^tinyint$/i,


Small int, big int, tinyint are all coverd by int$

Simplified int regex to /int$/i which covers all int types. Commit: 689a7ca

mathiasrw · 2025-12-22T22:50:39Z

src/15utility.js

+	// If no type specified, try to auto-convert if it looks like a number
+	if (!dbtypeid) {
+		if (alasql.options.csvStringToNumber && typeof value === 'string' && value.length > 0) {
+			var trimmed = value.trim();


Please avoid trim here.

Removed trim() call, now using direct comparison value == +value. Commit: 689a7ca

mathiasrw · 2025-12-22T22:51:16Z

src/15utility.js

+			var lower = value.toLowerCase();
+			return lower === 'true' || lower === '1' || lower === 'yes';


Can we please use a regex here instead?

Replaced toLowerCase + string comparisons with single regex: /^(true|1|yes)$/i.test(value). Commit: 689a7ca

…or boolean matching Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

mathiasrw · 2025-12-23T05:39:14Z

src/15utility.js

+ */
+var reTypeConverter = {
+	str: /string|char$|text/i,
+	int: /int$/i,


What about INTEGER? It needs. ^int aswell

mathiasrw · 2025-12-23T05:50:34Z

src/15utility.js

+ * @param {string} dbtypeid - The database type (INT, FLOAT, STRING, etc.)
+ * @return {*} The converted value
+ */
+utils.convertValueToType = function (value, dbtypeid) {


please rename to "typeConverter" so its aligned with reTypeConverter

Initial plan

5411079

Copilot AI assigned Copilot and mathiasrw Dec 22, 2025

Copilot started work on behalf of mathiasrw December 22, 2025 02:41 View session

Add csvStringToNumber option to control CSV type conversion

8ae503a

Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix wrong datatype issue after reading from CSV~~ Add csvStringToNumber option to control automatic string-to-number conversion in CSV imports Dec 22, 2025

Copilot AI requested a review from mathiasrw December 22, 2025 02:55

Copilot finished work on behalf of mathiasrw December 22, 2025 02:55

mathiasrw requested changes Dec 22, 2025

View reviewed changes

src/84from.js Show resolved Hide resolved

test/test-csv-string-type.js Outdated Show resolved Hide resolved

Copilot started work on behalf of mathiasrw December 22, 2025 05:57 View session

Copilot AI and others added 2 commits December 22, 2025 06:01

Refactor tests to use assert.deepEqual for complete object validation

1757841

Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

05448ca

Copilot AI requested a review from mathiasrw December 22, 2025 06:04

Copilot finished work on behalf of mathiasrw December 22, 2025 06:04

Copilot started work on behalf of mathiasrw December 22, 2025 10:01 View session

Implement column-type-aware conversion: keep CSV as strings, convert …

588303a

…at INSERT based on table definitions Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

Copilot AI changed the title ~~Add csvStringToNumber option to control automatic string-to-number conversion in CSV imports~~ Implement column-type-aware CSV import: defer type conversion to INSERT based on table definitions Dec 22, 2025

Copilot finished work on behalf of mathiasrw December 22, 2025 10:58

mathiasrw requested changes Dec 22, 2025

View reviewed changes

src/15utility.js Outdated Show resolved Hide resolved

src/40select.js Show resolved Hide resolved

test/test-csv-string-type.js Show resolved Hide resolved

Copilot started work on behalf of mathiasrw December 22, 2025 12:47 View session

Refactor: simplify type converter with regex patterns, streamline SEL…

06abb2d

…ECT INTO, add comprehensive edge case tests Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

Copilot AI requested a review from mathiasrw December 22, 2025 12:52

Copilot finished work on behalf of mathiasrw December 22, 2025 12:52

mathiasrw requested changes Dec 22, 2025

View reviewed changes

test/test-csv-string-type.js Show resolved Hide resolved

Copilot started work on behalf of mathiasrw December 22, 2025 15:41 View session

Remove individual property assertions, keep only assert.deepEqual for…

95707b5

… complete object validation Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

Copilot AI requested a review from mathiasrw December 22, 2025 15:45

Copilot finished work on behalf of mathiasrw December 22, 2025 15:45

mathiasrw requested changes Dec 22, 2025

View reviewed changes

Copilot started work on behalf of mathiasrw December 22, 2025 22:52 View session

Refactor type converter: simplify int regex, remove trim, use regex f…

689a7ca

…or boolean matching Co-authored-by: mathiasrw <1063454+mathiasrw@users.noreply.github.com>

Copilot AI requested a review from mathiasrw December 22, 2025 22:56

Copilot finished work on behalf of mathiasrw December 22, 2025 22:56

mathiasrw requested changes Dec 23, 2025

View reviewed changes

Copilot AI requested a review from mathiasrw December 23, 2025 05:50

		var lower = value.toLowerCase();
		return lower === 'true' \|\| lower === '1' \|\| lower === 'yes';

Implement column-type-aware CSV import: defer type conversion to INSERT based on table definitions #2403

Are you sure you want to change the base?

Implement column-type-aware CSV import: defer type conversion to INSERT based on table definitions #2403

Conversation

Copilot AI commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes Made

Type Converter Optimizations (src/15utility.js)

Testing Results

Comments on the Issue (you are @copilot in this section)

Uh oh!

Uh oh!

Uh oh!

mathiasrw commented Dec 22, 2025

Uh oh!

mathiasrw commented Dec 22, 2025

Uh oh!

Copilot AI commented Dec 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mathiasrw Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

mathiasrw Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

mathiasrw Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

mathiasrw Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

mathiasrw Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Dec 22, 2025 •

edited

Loading