Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2585 csv regression fix #2847

Merged
merged 3 commits into from
Mar 23, 2015
Merged

2585 csv regression fix #2847

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3bc7d06
Provide meaningful error for encoding error #2585
Mar 20, 2015
b1f4610
Tests for line breaks and broken encoding #2585
Mar 20, 2015
4fc1ee6
Deactivate test until clinker is updated #2585
Mar 20, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion services/importer/lib/importer/csv_normalizer.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ def normalize(temporary_filepath)
temporary_csv.close

@delimiter = OUTPUT_DELIMITER
rescue ArgumentError
rescue ArgumentError, Encoding::UndefinedConversionError
raise EncodingDetectionError
end

Expand Down
19 changes: 18 additions & 1 deletion services/importer/spec/acceptance/csv_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,9 +149,26 @@
SELECT count(*)
FROM #{result.schema}.#{result.table_name}
AS count
}].first.fetch(:count).should eq 7
}].first.fetch(:count).should eq 5
end

it 'refuses to import csv with broken encoding' do
pending "Need to update clinker to match production's ruby and stdlib"
filepath = path_to('broken_encoding.csv')
downloader = Downloader.new(filepath)
runner = Runner.new({
pg: @pg_options,
downloader: downloader,
log: CartoDB::Importer2::Doubles::Log.new,
user: CartoDB::Importer2::Doubles::User.new
})
runner.run

result = runner.results.first
runner.results.first.error_code.should eq CartoDB::Importer2::ERRORS_MAP[EncodingDetectionError]
end


it 'displays a specific error message for a file with too many columns' do
runner = runner_with_fixture('too_many_columns.csv')
runner.run
Expand Down
8 changes: 8 additions & 0 deletions services/importer/spec/fixtures/broken_encoding.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
State,County,Location,lng,lat,Metal Level,Issuer Name,Plan Marketing Name,Plan Type,Rating Area,Premium Adult Individual Age 27,Premium Adult Individual Age 50*,Premium Family,Premium Single Parent Family,Premium Couple*,Premium Child,Display
MI,ALCONA,alcona mi,-83.128974,44.683627,Catastrophic,Blue Cross Blue Shield of Huesca,Blue Cross� Premier Value,PPO,Rating Area 15,131.8,224.61,445.2,302.46,321.44,79.86,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Catastrophic,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Value,PPO,Rating Area 15,131.8,224.61,445.2,302.46,321.44,79.86,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Bronze,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Bronze,PPO,Rating Area 15,172.99,294.82,584.34,396.99,421.92,104.82,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Silver,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Silver,PPO,Rating Area 15,227.31,387.38,767.82,521.64,554.4,137.73,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Gold,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Gold,PPO,Rating Area 15,274.62,468,927.64,630.22,669.78,166.4,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Silver,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Silver- a Multi-State Plan,PPO,Rating Area 15,243.44,414.87,822.3,558.65,593.74,147.5,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Gold,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Gold- a Multi-State Plan,PPO,Rating Area 15,291.21,496.28,983.66,668.28,710.24,176.45,Yes
15 changes: 8 additions & 7 deletions services/importer/spec/fixtures/in_cell_line_breaks.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
State,County,Location,lng,lat,Metal Level,Issuer Name,Plan Marketing Name,Plan Type,Rating Area,Premium Adult Individual Age 27,Premium Adult Individual Age 50*,Premium Family,Premium Single Parent Family,Premium Couple*,Premium Child,Display
MI,ALCONA,alcona mi,-83.128974,44.683627,Catastrophic,Blue Cross Blue Shield of Huesca,Blue Cross� Premier Value,PPO,Rating Area 15,131.8,224.61,445.2,302.46,321.44,79.86,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Catastrophic,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Value,PPO,Rating Area 15,131.8,224.61,445.2,302.46,321.44,79.86,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Bronze,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Bronze,PPO,Rating Area 15,172.99,294.82,584.34,396.99,421.92,104.82,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Silver,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Silver,PPO,Rating Area 15,227.31,387.38,767.82,521.64,554.4,137.73,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Gold,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Premier Gold,PPO,Rating Area 15,274.62,468,927.64,630.22,669.78,166.4,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Silver,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Silver- a Multi-State Plan,PPO,Rating Area 15,243.44,414.87,822.3,558.65,593.74,147.5,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Gold,Blue Cross Blue Shield of Michigan,Blue CrossՊ� Gold- a Multi-State Plan,PPO,Rating Area 15,291.21,496.28,983.66,668.28,710.24,176.45,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Catastrophic,Blue Cross Blue Shield of Huesca,"Blue Cross
Premier Value",PPO,Rating Area 15,131.8,224.61,445.2,302.46,321.44,79.86,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Catastrophic,Blue Cross Blue Shield of Michigan,"Blue Cross Premier Value",PPO,Rating Area 15,131.8,224.61,445.2,302.46,321.44,79.86,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Bronze,Blue Cross Blue Shield of Michigan,"Blue Cross
Premier Bronze",PPO,Rating Area 15,172.99,294.82,584.34,396.99,421.92,104.82,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Silver,Blue Cross Blue Shield of Michigan,"Blue Cross
Premier Silver",PPO,Rating Area 15,227.31,387.38,767.82,521.64,554.4,137.73,Yes
MI,ALCONA,alcona mi,-83.128974,44.683627,Gold,Blue Cross Blue Shield of Michigan,Blue Cross Premier Gold,PPO,Rating Area 15,274.62,468,927.64,630.22,669.78,166.4,Yes
Expand Down