:convert_values_to_numeric_unless_leading_zeros drops leading zeros

[dmolesUC]

Steps to reproduce:

Given a file called data.csv with the following:

id	start	end
40456	0900	2100
40881	0900	0200
74581	0900	1645
98359	0900	1800
141635	0900	1645
178431	0800	0200
220975	1300	1700
266212	0900	1800
276688	0900	2200
288836	0900	1700

Run the following script:

#!/usr/bin/env ruby

require 'smarter_csv'

path = File.expand_path('data.csv', __dir__)
data = SmarterCSV.process(
  path,
  {
    col_sep: "\t",
    hash_transformations: [:remove_blank_values, :convert_values_to_numeric_unless_leading_zeros]
  }
)

data.each do |row|
  s, e = row[:start], row[:end]
  puts "start: #{s} (#{s.class})\tend: #{e} (#{e.class})"
end

Expected

• Values are strings
• Leading zeroes are preserved

Actual

start: 900 (Integer)	end: 2100 (Integer)
start: 900 (Integer)	end: 200 (Integer)
start: 900 (Integer)	end: 1645 (Integer)
start: 900 (Integer)	end: 1800 (Integer)
start: 900 (Integer)	end: 1645 (Integer)
start: 800 (Integer)	end: 200 (Integer)
start: 1300 (Integer)	end: 1700 (Integer)
start: 900 (Integer)	end: 1800 (Integer)
start: 900 (Integer)	end: 2200 (Integer)
start: 900 (Integer)	end: 1700 (Integer)

[melbreaker]

Which version of the library are you using? I'm using version 1.2.
Adding :convert_values_to_numeric => false in options solve the issue for me.

[dmolesUC]

1.2.6. If I set :convert_values_to_numeric => false that does stop these values from being converted to numeric, but it also stops other values from being converted to numeric as well.

As a workaround I can unset :convert_values_to_numeric_unless_leading_zeros and set convert_values_to_numeric: { except: [:start, :end] }:

#!/usr/bin/env ruby

require 'smarter_csv'

path = File.expand_path('data.csv', __dir__)
data = SmarterCSV.process(
  path,
  {
    col_sep: "\t",
    convert_values_to_numeric: { except: [:start, :end] },
    hash_transformations: [:remove_blank_values ]
  }
)

data.each do |row|
  i, s, e = row[:id], row[:start], row[:end]
  puts "id: #{i} (#{i.class})\tstart: #{s} (#{s.class})\tend: #{e} (#{e.class})"
end

# => prints:

id: 40456 (Integer)	start: 0900 (String)	end: 2100 (String)
id: 40881 (Integer)	start: 0900 (String)	end: 0200 (String)
id: 74581 (Integer)	start: 0900 (String)	end: 1645 (String)
id: 98359 (Integer)	start: 0900 (String)	end: 1800 (String)
id: 141635 (Integer)	start: 0900 (String)	end: 1645 (String)
id: 178431 (Integer)	start: 0800 (String)	end: 0200 (String)
id: 220975 (Integer)	start: 1300 (String)	end: 1700 (String)
id: 266212 (Integer)	start: 0900 (String)	end: 1800 (String)
id: 276688 (Integer)	start: 0900 (String)	end: 2200 (String)
id: 288836 (Integer)	start: 0900 (String)	end: 1700 (String)

Which arguably is better semantics.

But isn't this what :convert_values_to_numeric_unless_leading_zeros is supposed to do?

[fuzzygroup]

I had the same issue. I don't think this is fixed. The work around did work tho.

[tilo]

the solution for 1.x is the way to go: convert_values_to_numeric: { except: [:start, :end] }

This is ear-marked for 2.0

[9mm]

how do we do this in 2.0? i dont want any conversions at all, i just want all strings

[tilo]

e.g.

data = SmarterCSV.process('/tmp/zip.csv',  convert_values_to_numeric: { except: [:zip] })
 => [{:zip=>"00480"}, {:zip=>"51903"}, {:zip=>"12354"}, {:zip=>"02343"}]

[tilo]

@fuzzygroup it is now documented here: https://github.com/tilo/smarter_csv/blob/main/docs/data_transformations.md#convert-values-to-numeric