New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSVDataSet does not read UTF-8 files when file.encoding is UTF-8 #2512
Comments
Jacob Zwiers (migrated from Bugzilla): Created attachment 51645-testcases.patch: Test cases to expose bug. Run with file.encoding=UTF-8 51645-testcases.patchIndex: test/src/org/apache/jmeter/config/TestCVSDataSet.java
===================================================================
--- test/src/org/apache/jmeter/config/TestCVSDataSet.java (revision 1155546)
+++ test/src/org/apache/jmeter/config/TestCVSDataSet.java (working copy)
@@ -92,7 +92,41 @@
assertEquals("b1",threadVars.get("b"));
assertEquals("c1",threadVars.get("c"));
}
+
+ public void testutf8() throws Exception {
+
+ CSVDataSet csv = new CSVDataSet();
+ csv.setFilename(findTestPath("testfiles/testutf8.csv"));
+ csv.setVariableNames("a,b,c,d");
+ csv.setDelimiter(",");
+ csv.setQuotedData( true );
+ csv.setFileEncoding( "UTF-8" );
+
+ csv.iterationStart(null);
+ assertEquals("a1",threadVars.get("a"));
+ assertEquals("b1",threadVars.get("b"));
+ assertEquals("\u00e71",threadVars.get("c"));
+ assertEquals("d1",threadVars.get("d"));
+ csv.iterationStart(null);
+ assertEquals("a2",threadVars.get("a"));
+ assertEquals("b2",threadVars.get("b"));
+ assertEquals("\u00e72",threadVars.get("c"));
+ assertEquals("d2",threadVars.get("d"));
+
+ csv.iterationStart(null);
+ assertEquals("a3",threadVars.get("a"));
+ assertEquals("b3",threadVars.get("b"));
+ assertEquals("\u00e73",threadVars.get("c"));
+ assertEquals("d3",threadVars.get("d"));
+
+ csv.iterationStart(null);
+ assertEquals("a4",threadVars.get("a"));
+ assertEquals("b4",threadVars.get("b"));
+ assertEquals("\u00e74",threadVars.get("c"));
+ assertEquals("d4",threadVars.get("d"));
+ }
+
// Test CSV file with a header line
public void testHeaderOpen(){
CSVDataSet csv = new CSVDataSet();
Index: test/src/org/apache/jmeter/save/TestCSVSaveService.java
===================================================================
--- test/src/org/apache/jmeter/save/TestCSVSaveService.java (revision 1155546)
+++ test/src/org/apache/jmeter/save/TestCSVSaveService.java (working copy)
@@ -60,6 +60,10 @@
checkSplitString("a,bc,,", ',', new String[]{"a","bc","",""});
checkSplitString("a,,,", ',', new String[]{"a","","",""});
checkSplitString("a,bc,d,\n",',', new String[]{"a","bc","d",""});
+
+ // \u00e7 = LATIN SMALL LETTER C WITH CEDILLA
+ // \u00e9 = LATIN SMALL LETTER E WITH ACUTE
+ checkSplitString("a,b\u00e7,d,\u00e9", ',', new String[]{"a","b\u00e7","d","\u00e9"});
}
public void testSplitQuoted() throws Exception {
@@ -75,6 +79,10 @@
checkSplitString("a,bc,d,", ',', new String[]{"a","bc","d",""});
checkSplitString("a,bc,d,\"\"", ',', new String[]{"a","bc","d",""});
checkSplitString("a,bc,d,\"\"\n",',', new String[]{"a","bc","d",""});
+
+ // \u00e7 = LATIN SMALL LETTER C WITH CEDILLA
+ // \u00e9 = LATIN SMALL LETTER E WITH ACUTE
+ checkSplitString("\"a\",\"b\u00e7\",\"d\",\"\u00e9\"", ',', new String[]{"a","b\u00e7","d","\u00e9"});
}
public void testSplitBadQuote() throws Exception { |
Jacob Zwiers (migrated from Bugzilla): Created attachment testutf8.csv: .csv file for previous test patch |
Sebb (migrated from Bugzilla): Patch applied: URL: http://svn.apache.org/viewvc?rev=1156416&view=rev Added: |
Jacob Zwiers (Bug 51645):
CSV Data Sets which are encoded in UTF-8 do not work on platforms where the default file.encoding is UTF-8.
UTF-8 is used to illustrate here, but this would presumably apply to other non-8bit character sets as well.
Reason: The use of ByteArrayOutputStream in the CSVSaveService.csvReadFile() method. Specifically, the boas.write(ch) call is implemented (internally in ByteArrayOutputStream) with a cast to the byte primitive type ( buf[count] = (byte)b; in my JVM).
Later, the ByteArrayOutputStream is interpreted according to the platform default (via baos.toString()) and if the content of the array are then interpreted according to the platform's default char set. If that charset (eg. ISO-8859-1) is 8-bit, everything is fine. However, unpredictable results/unmapped chars result for other charsets (like UTF-8).
For example, the character \u0027 (LATIN SMALL LETTER C WITH CEDILLA) with decimal code point 231. When put into boas, it becomes (7 bit signed) -25. When converted via toString() with UTF-8 as the default char set, the value is not recognized as a valid code point and the value \ufffd (decimal code point 65533 == Unicodes "REPLACEMENT CHARACTER" placeholder) is placed in the return string instead.
Fix: patche attached. Simply replace ByteArrayOutputStream with CharArrayWriter and the UTF-8 files work regardless of the value for file.encoding.
Created attachment CSVSaveService.java.patch: Patch to fix issue. Variable not renamed to show just a matter of replacing class.
CSVSaveService.java.patch
Severity: major
OS: All
The text was updated successfully, but these errors were encountered: