Removing processing of extraneous blank fields and enlarged arrays

Author: ccrook

resources/context_help/QgsDelimitedTextSourceSelect

@@ -35,7 +35,9 @@ be double, otherwise the type will be text.
 QGIS can also read the types from an OGR CSV driver compatible "csvt" file.  
 This is a file alongside the data file, but with a "t" appended to the file name. 
 The file should just contain one line which lists the type of each field. 
-Valid types are "integer", "real", "string", "date", "time", and "datetime". The date, time, and datetime types are treated as strings in QGIS.
+Valid types are "integer", "long", "longlong", "real", 
+"string", "date", "time", and "datetime". 
+The date, time, and datetime types are treated as strings by the delimited text provider.
 Each type may be followed by a width and precision, for example "real(10.4)".
 The list of types are separated by commas, regardless of the delimiter used in the data file.  An
 example of a valid format file would be:
@@ -215,9 +217,11 @@ feature id to each record which is the line number in the source file on which
 the record starts.  
 </p>
 <p>
-Each attribute also has a data type, one of string (text), integer, or real number.
+Each attribute also has a data type, one of string (text), integer, longlong,
+or real number.
 The data type is inferred from the content of the fields - if every non blank value
-is a valid integer then the type is integer, otherwise if it is a valid real
+is a valid integer then the type is integer, otherwise if it is a valid long long
+nubmer then the type is longlong, otherwise if it is a valid real
 number then the type is real, otherwise the type is string.  Note that this is
 based on the content of the fields - quoting fields does not change the way they
 are interpreted.

src/providers/delimitedtext/qgsdelimitedtextprovider.cpp

@@ -562,29 +562,46 @@ void QgsDelimitedTextProvider::scanFile( bool buildIndexes )
     {
 
       QString &value = parts[i];
-      if ( value.isEmpty() ) {
+      // Ignore empty fields - spreadsheet generated CSV files often 
+      // have random empty fields at the end of a row
+      if ( value.isEmpty() )
+        continue;
+
+      // Expand the columns to include this non empty field if necessary
+
+      while ( couldBeInt.size() <= i )
+      {
         isEmpty.append( true );
         couldBeInt.append( false );
         couldBeLongLong.append( false );
         couldBeDouble.append( false );
-        continue;
       }
-      // try to convert attribute values to integer, long and double
-      isEmpty.append( false );
-      // try all possible formats
-      couldBeInt.append( true );
-      couldBeLongLong.append( true );
-      couldBeDouble.append( true );
+
+      // If this column has been empty so far then initiallize it
+      // for possible types
+
+      if ( isEmpty[i] )
+      {
+        isEmpty[i] = false;
+        couldBeInt[i] = true;
+        couldBeLongLong[i] = true;
+        couldBeDouble[i] = true;
+      }
+
+      // Now test for still valid possible types for the field
+      // Types are possible until first record which cannot be parsed
+
       if ( couldBeInt[i] )
       {
         value.toInt( &couldBeInt[i] );
       }
 
-      if ( couldBeLongLong[i] )
+      if ( couldBeLongLong[i] && ! couldBeInt[i] )
       {
         value.toLongLong( &couldBeLongLong[i] );
       }
-      if ( couldBeDouble[i] )
+
+      if ( couldBeDouble[i] && ! couldBeLongLong[i] )
       {
         if ( ! mDecimalPoint.isEmpty() )
         {
@@ -627,7 +644,7 @@ void QgsDelimitedTextProvider::scanFile( bool buildIndexes )
         fieldType = QVariant::LongLong; //QVariant doesn't support long
         typeName = "longlong";
       }
-      else if ( csvtTypes[i] == "real" )
+      else if ( csvtTypes[i] == "real" || csvtTypes[i] == "double" )
       {
         fieldType = QVariant::Double;
         typeName = "double";