Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First Name and Nickname parsing #84

Merged
merged 5 commits into from Apr 23, 2019

Conversation

Projects
None yet
2 participants
@mscarey
Copy link
Contributor

commented Apr 18, 2019

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Description

Selects the deceased's first name as their given name, rather than the last name prior to their surname.

A deceased's nickname was being recorded as his given name.

I added tests of apd.parse_name and apd.parse_deceased_field with this name, but I wasn't able to figure out how to test whether the correct data was being passed in to apd.parse_deceased_field.

Checklist:

  • [] I have updated the documentation accordingly
  • I have written unit tests

Fixes: #74

mscarey added some commits Apr 17, 2019

use first item of deceased_field for First Name
Changed the part of parse_deceased_field that removed some items and then used the last remaining item as the First Name. I wasn't able to verify that the issue (#74, recording a nickname as the first name) is fixed in the cli.

@rgreinho rgreinho self-requested a review Apr 18, 2019

@rgreinho
Copy link
Member

left a comment

Good job with this patch @mscarey!

The linting/formatting issues should be resolved if you run the make format command.

Regarding the tests, if you add a scenario which would trigger an exception, that should satisfy the coverage requirement.

@rgreinho

This comment has been minimized.

Copy link
Member

commented Apr 18, 2019

Here is a patch that would help with the tests/coverage:

diff --git a/scrapd/core/apd.py b/scrapd/core/apd.py
index f15c717..8bbaec0 100644
--- a/scrapd/core/apd.py
+++ b/scrapd/core/apd.py
@@ -278,7 +278,7 @@ def parse_name(name):
     try:
         d["last"] = name[-1].replace(',', '')
         d["first"] = name[0].replace(',', '')
-    except IndexError:
+    except (IndexError, TypeError):
         pass
     return d
 
@@ -326,8 +326,8 @@ def parse_deceased_field(deceased_field):
         pass
 
     name = parse_name(fleg)
-    d[Fields.LAST_NAME] = name.get("last")
-    d[Fields.FIRST_NAME] = name.get("first")
+    d[Fields.LAST_NAME] = name.get('last', '')
+    d[Fields.FIRST_NAME] = name.get('first', '')
     return d
 
 
diff --git a/tests/core/test_apd.py b/tests/core/test_apd.py
index 43dcad0..2b9a6d1 100644
--- a/tests/core/test_apd.py
+++ b/tests/core/test_apd.py
@@ -243,12 +243,16 @@ def test_parse_deceased_field():
         'first': 'Delta',
         'last': 'Olin'
     }),
+    (None, {
+        'first': None,
+        'last': None
+    }),
 ))
 def test_parse_name(name, expected):
     """Ensure parser finds the first and last name given the full name."""
     parsed = apd.parse_name(name)
-    assert parsed["first"] == expected["first"]
-    assert parsed["last"] == expected["last"]
+    assert parsed.get("first") == expected["first"]
+    assert parsed.get("last") == expected["last"]
 
 
 def test_extract_traffic_fatalities_page_details_link_01():
@rgreinho

This comment has been minimized.

Copy link
Member

commented Apr 18, 2019

I just ran ScrAPD from your branch, and it does solve A LOT of parsing problems (see below)! 61 entries out of 148 (41% of the current data set!)

The only one it did not catch is https://austintexas.gov/news/traffic-fatality-26-2.

diff --git a/scrapd.all.json b/scrapd.all.json
index 6503d5c..3d93fe5 100644
--- a/scrapd.all.json
+++ b/scrapd.all.json
@@ -63,7 +63,7 @@
     "Date": "03/29/2019",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "12",
-    "First Name": "Cardenas",
+    "First Name": "Carlos",
     "Gender": "male",
     "Last Name": "Jr.",
     "Link": "http://austintexas.gov/news/traffic-fatality-12-4",
@@ -78,7 +78,7 @@
     "Date": "03/28/2019",
     "Ethnicity": "White",
     "Fatal crashes this year": "11",
-    "First Name": "Rae",
+    "First Name": "Jessica",
     "Gender": "female",
     "Last Name": "Saathoff",
     "Link": "http://austintexas.gov/news/traffic-fatality-11-4",
@@ -93,7 +93,7 @@
     "Date": "02/27/2019",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "10",
-    "First Name": "Ni\u00f1o",
+    "First Name": "Javier",
     "Gender": "male",
     "Last Name": "Esparza",
     "Link": "http://austintexas.gov/news/traffic-fatality-10-4",
@@ -108,7 +108,7 @@
     "Date": "02/21/2019",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "9",
-    "First Name": "\u201cRudy\u201d",
+    "First Name": "Rosbel",
     "Gender": "male",
     "Last Name": "Tamez",
     "Link": "http://austintexas.gov/news/traffic-fatality-9-4",
@@ -138,7 +138,7 @@
     "Date": "02/09/2019",
     "Ethnicity": "Black",
     "Fatal crashes this year": "7",
-    "First Name": "Zion",
+    "First Name": "Messiah",
     "Gender": "male",
     "Last Name": "Mouton",
     "Link": "http://austintexas.gov/news/traffic-fatality-7-4",
@@ -153,7 +153,7 @@
     "Date": "02/06/2019",
     "Ethnicity": "White",
     "Fatal crashes this year": "6",
-    "First Name": "James",
+    "First Name": "Trevor",
     "Gender": "male",
     "Last Name": "Ralston",
     "Link": "http://austintexas.gov/news/traffic-fatality-6-6",
@@ -213,7 +213,7 @@
     "Date": "01/15/2019",
     "Ethnicity": "White",
     "Fatal crashes this year": "1",
-    "First Name": "Hilburn",
+    "First Name": "David",
     "Gender": "male",
     "Last Name": "Sell",
     "Link": "http://austintexas.gov/news/traffic-fatality-1-4",
@@ -303,7 +303,7 @@
     "Date": "12/01/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "70",
-    "First Name": "Jack",
+    "First Name": "Billy",
     "Gender": "male",
     "Last Name": "Dickey",
     "Link": "http://austintexas.gov/news/traffic-fatality-70-2",
@@ -333,7 +333,7 @@
     "Date": "11/22/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "67",
-    "First Name": "Lucille",
+    "First Name": "Mary",
     "Gender": "female",
     "Last Name": "Ochoa",
     "Link": "http://austintexas.gov/news/traffic-fatality-67-3",
@@ -393,7 +393,7 @@
     "Date": "11/03/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "63",
-    "First Name": "Ross",
+    "First Name": "Aaron",
     "Gender": "male",
     "Last Name": "Weir",
     "Link": "http://austintexas.gov/news/traffic-fatality-63-3",
@@ -417,7 +417,7 @@
     "Date": "10/30/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "60",
-    "First Name": "Marie",
+    "First Name": "Stefanie",
     "Gender": "female",
     "Last Name": "Torres",
     "Link": "http://austintexas.gov/news/traffic-fatality-60-3",
@@ -432,7 +432,7 @@
     "Date": "10/28/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "61",
-    "First Name": "Alexander",
+    "First Name": "Anthony",
     "Gender": "male",
     "Last Name": "Zangara",
     "Link": "http://austintexas.gov/news/traffic-fatality-61-3",
@@ -531,7 +531,7 @@
     "Date": "10/03/2018",
     "Ethnicity": "Black",
     "Fatal crashes this year": "53",
-    "First Name": "Duron",
+    "First Name": "Michael",
     "Gender": "male",
     "Last Name": "Green",
     "Link": "http://austintexas.gov/news/traffic-fatality-53-5",
@@ -570,7 +570,7 @@
     "Date": "09/11/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "50",
-    "First Name": "Javier",
+    "First Name": "Ramon",
     "Gender": "male",
     "Last Name": "Pesina",
     "Link": "http://austintexas.gov/news/traffic-fatality-50-2",
@@ -609,7 +609,7 @@
     "Date": "09/02/2018",
     "Ethnicity": "Black",
     "Fatal crashes this year": "47",
-    "First Name": "Anthony",
+    "First Name": "Michael",
     "Gender": "male",
     "Last Name": "Reed-Harper",
     "Link": "http://austintexas.gov/news/traffic-fatality-47-5",
@@ -633,7 +633,7 @@
     "Date": "08/21/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "46",
-    "First Name": "Taylor",
+    "First Name": "Kendall",
     "Gender": "female",
     "Last Name": "Dezelle",
     "Link": "http://austintexas.gov/news/traffic-fatality-46-5",
@@ -648,7 +648,7 @@
     "Date": "08/25/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "44",
-    "First Name": "Rubio",
+    "First Name": "Juvenal",
     "Gender": "male",
     "Last Name": "Martinez",
     "Link": "http://austintexas.gov/news/traffic-fatality-44-4",
@@ -663,7 +663,7 @@
     "Date": "08/15/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "45",
-    "First Name": "Edward",
+    "First Name": "Thomas",
     "Gender": "male",
     "Last Name": "Craig",
     "Link": "http://austintexas.gov/news/traffic-fatality-45-4",
@@ -678,7 +678,7 @@
     "Date": "08/21/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "42",
-    "First Name": "Wayne",
+    "First Name": "Jarrod",
     "Gender": "male",
     "Last Name": "Sowers",
     "Link": "http://austintexas.gov/news/traffic-fatality-42-3",
@@ -765,7 +765,7 @@
     "Date": "07/26/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "36",
-    "First Name": "Cristobal",
+    "First Name": "Luis",
     "Gender": "male",
     "Last Name": "Landeros",
     "Link": "http://austintexas.gov/news/traffic-fatality-36-2",
@@ -780,7 +780,7 @@
     "Date": "07/15/2018",
     "Ethnicity": "Black",
     "Fatal crashes this year": "35",
-    "First Name": "Ray",
+    "First Name": "Jacoby",
     "Gender": "male",
     "Last Name": "Bailey",
     "Link": "http://austintexas.gov/news/traffic-fatality-35-2",
@@ -795,7 +795,7 @@
     "Date": "07/23/2018",
     "Ethnicity": "Other",
     "Fatal crashes this year": "34",
-    "First Name": "Thomas",
+    "First Name": "Brandon",
     "Gender": "male",
     "Last Name": "Buentello",
     "Link": "http://austintexas.gov/news/traffic-fatality-34-3",
@@ -810,7 +810,7 @@
     "Date": "07/23/2018",
     "Ethnicity": "Other",
     "Fatal crashes this year": "33",
-    "First Name": "Gerald",
+    "First Name": "Scott",
     "Gender": "male",
     "Last Name": "Whiting",
     "Link": "http://austintexas.gov/news/traffic-fatality-33-4",
@@ -825,7 +825,7 @@
     "Date": "07/13/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "32",
-    "First Name": "Dean",
+    "First Name": "Troy",
     "Gender": "male",
     "Last Name": "Fisher",
     "Link": "http://austintexas.gov/news/traffic-fatality-32-4",
@@ -840,7 +840,7 @@
     "Date": "07/09/2018",
     "Ethnicity": "Asian",
     "Fatal crashes this year": "31",
-    "First Name": "Dineshbhai",
+    "First Name": "Pinkeshkumar",
     "Gender": "male",
     "Last Name": "Patel",
     "Link": "http://austintexas.gov/news/traffic-fatality-31-2",
@@ -900,7 +900,9 @@
     "Date": "06/10/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "26",
+    "First Name": "",
     "Gender": "male",
+    "Last Name": "",
     "Link": "http://austintexas.gov/news/traffic-fatality-26-2",
     "Location": "E US 290 and Harris Branch Parkway",
     "Notes": "Jonathan, Garcia-Pineda, The case is still being investigated. Anyone with information regarding this incident is asked to call the APD Vehicular Homicide Unit Detectives at (512) 974-6935. You can also submit tips by downloading APD\u2019s mobile app, Austin PD, for free on iPhone and Android. This is Austin\u2019s twenty-sixth fatal traffic crash of 2018, resulting in twenty-seven fatalities this year. At this time in 2017, there were twenty-eight fatal traffic crashes and thirty traffic fatalities. These statements are based on the initial assessment of the fatal crash and investigation is still pending. Fatality information may change.",
@@ -961,7 +963,7 @@
     "Date": "05/01/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "21",
-    "First Name": "Renae",
+    "First Name": "Marissa",
     "Gender": "female",
     "Last Name": "Rodriguez",
     "Link": "http://austintexas.gov/news/traffic-fatality-21-2",
@@ -1021,7 +1023,7 @@
     "Date": "04/11/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "17",
-    "First Name": "Andrew",
+    "First Name": "Afshin",
     "Gender": "male",
     "Last Name": "Sustaita",
     "Link": "http://austintexas.gov/news/traffic-fatality-17-3",
@@ -1045,7 +1047,7 @@
     "Date": "04/09/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "16",
-    "First Name": "Orduna",
+    "First Name": "Juan",
     "Gender": "male",
     "Last Name": "Luna",
     "Link": "http://austintexas.gov/news/traffic-fatality-16-3",
@@ -1084,7 +1086,7 @@
     "Date": "02/28/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "11",
-    "First Name": "Franco",
+    "First Name": "Daniel",
     "Gender": "male",
     "Last Name": "Tovar",
     "Link": "http://austintexas.gov/news/traffic-fatality-11-3",
@@ -1099,7 +1101,7 @@
     "Date": "03/25/2018",
     "Ethnicity": "Black",
     "Fatal crashes this year": "14",
-    "First Name": "Douglas",
+    "First Name": "Zachary",
     "Gender": "male",
     "Last Name": "McBride",
     "Link": "http://austintexas.gov/news/traffic-fatality-14-4",
@@ -1129,7 +1131,7 @@
     "Date": "02/17/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "9",
-    "First Name": "Reed",
+    "First Name": "Devon",
     "Gender": "male",
     "Last Name": "Gerald",
     "Link": "http://austintexas.gov/news/traffic-fatality-9-3",
@@ -1144,7 +1146,7 @@
     "Date": "02/06/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "8",
-    "First Name": "Antonio",
+    "First Name": "Mark",
     "Gender": "male",
     "Last Name": "Rodriguez",
     "Link": "http://austintexas.gov/news/traffic-fatality-8-3",
@@ -1175,7 +1177,7 @@
     "Date": "02/09/2018",
     "Ethnicity": "White",
     "Fatal crashes this year": "7",
-    "First Name": "Wayne",
+    "First Name": "Timothy",
     "Gender": "male",
     "Last Name": "Boykin",
     "Link": "http://austintexas.gov/news/traffic-fatality-7-3",
@@ -1190,7 +1192,7 @@
     "Date": "01/30/2018",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "5",
-    "First Name": "Rosa",
+    "First Name": "Alma",
     "Gender": "female",
     "Last Name": "Hernandez-Jaimes",
     "Link": "http://austintexas.gov/news/traffic-fatality-5-6",
@@ -1260,7 +1262,7 @@
     "Date": "12/21/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "71",
-    "First Name": "Carson",
+    "First Name": "Shane",
     "Gender": "male",
     "Last Name": "Goldstein",
     "Link": "http://austintexas.gov/news/traffic-fatality-71-0",
@@ -1275,7 +1277,7 @@
     "Date": "12/12/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "70",
-    "First Name": "Lance",
+    "First Name": "Robert",
     "Gender": "male",
     "Last Name": "Trewitt",
     "Link": "http://austintexas.gov/news/traffic-fatality-70-0",
@@ -1290,7 +1292,7 @@
     "Date": "12/05/2017",
     "Ethnicity": "Black",
     "Fatal crashes this year": "68",
-    "First Name": "Lee",
+    "First Name": "Deion",
     "Gender": "male",
     "Last Name": "Odom",
     "Link": "http://austintexas.gov/news/traffic-fatality-68-1",
@@ -1335,7 +1337,7 @@
     "Date": "12/03/2017",
     "Ethnicity": "Eastern",
     "Fatal crashes this year": "66",
-    "First Name": "Alqassab",
+    "First Name": "Sajjad",
     "Gender": "male",
     "Last Name": "Middle",
     "Link": "http://austintexas.gov/news/traffic-fatality-66-1",
@@ -1389,7 +1391,7 @@
     "Date": "11/23/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "62",
-    "First Name": "Ashley",
+    "First Name": "Miranda",
     "Gender": "female",
     "Last Name": "Stark",
     "Link": "http://austintexas.gov/news/traffic-fatality-62-1",
@@ -1404,7 +1406,7 @@
     "Date": "11/20/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "61",
-    "First Name": "Joseph",
+    "First Name": "Christopher",
     "Gender": "male",
     "Last Name": "Gorman",
     "Link": "http://austintexas.gov/news/traffic-fatality-61-1",
@@ -1452,7 +1454,7 @@
     "Date": "11/05/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "57",
-    "First Name": "Ryan",
+    "First Name": "Zachary",
     "Gender": "male",
     "Last Name": "Alexander-Ball",
     "Link": "http://austintexas.gov/news/traffic-fatality-57-1",
@@ -1467,7 +1469,7 @@
     "Date": "10/31/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "56",
-    "First Name": "Mauricio",
+    "First Name": "Marcos",
     "Gender": "male",
     "Last Name": "Morales",
     "Link": "http://austintexas.gov/news/traffic-fatality-56-2",
@@ -1539,7 +1541,7 @@
     "Date": "09/02/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "47",
-    "First Name": "(Josie)",
+    "First Name": "Josefa",
     "Gender": "female",
     "Last Name": "Torres",
     "Link": "http://austintexas.gov/news/traffic-fatality-47-3",
@@ -1554,7 +1556,7 @@
     "Date": "09/07/2017",
     "Ethnicity": "Black",
     "Fatal crashes this year": "46",
-    "First Name": "Adreon",
+    "First Name": "Sterlin",
     "Gender": "male",
     "Last Name": "Milam",
     "Link": "http://austintexas.gov/news/traffic-fatality-46-3",
@@ -1599,7 +1601,7 @@
     "Date": "09/03/2017",
     "Ethnicity": "Black",
     "Fatal crashes this year": "42",
-    "First Name": "Johnson",
+    "First Name": "Spencer",
     "Gender": "male",
     "Last Name": "II",
     "Link": "http://austintexas.gov/news/traffic-fatality-42-2",
@@ -1614,7 +1616,7 @@
     "Date": "09/04/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "43",
-    "First Name": "Stacy",
+    "First Name": "Richard",
     "Gender": "male",
     "Last Name": "Stader",
     "Link": "http://austintexas.gov/news/traffic-fatality-43-2",
@@ -1629,7 +1631,7 @@
     "Date": "08/23/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "41",
-    "First Name": "Isaac",
+    "First Name": "Matthew",
     "Gender": "male",
     "Last Name": "Ramos",
     "Link": "http://austintexas.gov/news/traffic-fatality-41-2",
@@ -1644,7 +1646,7 @@
     "Date": "08/14/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "40",
-    "First Name": "Roderick",
+    "First Name": "Alexander",
     "Gender": "male",
     "Last Name": "Hopkins",
     "Link": "http://austintexas.gov/news/traffic-fatality-40-2",
@@ -1689,7 +1691,7 @@
     "Date": "08/05/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "37",
-    "First Name": "Garay",
+    "First Name": "Melvin",
     "Gender": "male",
     "Last Name": "Chacon",
     "Link": "http://austintexas.gov/news/traffic-fatality-37-2",
@@ -1704,7 +1706,7 @@
     "Date": "08/03/2017",
     "Ethnicity": "Black",
     "Fatal crashes this year": "36",
-    "First Name": "Monique",
+    "First Name": "Kenyae",
     "Gender": "female",
     "Last Name": "Dixon",
     "Link": "http://austintexas.gov/news/traffic-fatality-36-1",
@@ -1719,7 +1721,7 @@
     "Date": "08/02/2017",
     "Ethnicity": "Asian",
     "Fatal crashes this year": "35",
-    "First Name": "Allen",
+    "First Name": "Tracy",
     "Gender": "male",
     "Last Name": "Dam",
     "Link": "http://austintexas.gov/news/traffic-fatality-35-1",
@@ -1749,7 +1751,7 @@
     "Date": "07/17/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "33",
-    "First Name": "James",
+    "First Name": "Michael",
     "Gender": "male",
     "Last Name": "Trujillo",
     "Link": "http://austintexas.gov/news/traffic-fatality-33-3",
@@ -1764,7 +1766,7 @@
     "Date": "07/13/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "32",
-    "First Name": "David",
+    "First Name": "Robert",
     "Gender": "male",
     "Last Name": "Register",
     "Link": "http://austintexas.gov/news/traffic-fatality-32-3",
@@ -1779,7 +1781,7 @@
     "Date": "07/08/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "31",
-    "First Name": "Joseph",
+    "First Name": "Daniel",
     "Gender": "male",
     "Last Name": "Rayos",
     "Link": "http://austintexas.gov/news/traffic-fatality-31-1",
@@ -1794,7 +1796,7 @@
     "Date": "06/10/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "29",
-    "First Name": "R.",
+    "First Name": "Daniel",
     "Gender": "male",
     "Last Name": "Sullins",
     "Link": "http://austintexas.gov/news/traffic-fatality-29-1",
@@ -1809,7 +1811,7 @@
     "Date": "06/05/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "28",
-    "First Name": "Juan",
+    "First Name": "Garcia",
     "Gender": "male",
     "Last Name": "Narciso",
     "Link": "http://austintexas.gov/news/traffic-fatality-28-1",
@@ -1824,7 +1826,7 @@
     "Date": "05/30/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "27",
-    "First Name": "Emmanuel",
+    "First Name": "Ramirez",
     "Gender": "male",
     "Last Name": "Castro",
     "Link": "http://austintexas.gov/news/traffic-fatality-27-2",
@@ -1839,7 +1841,7 @@
     "Date": "05/01/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "26",
-    "First Name": "Deborah",
+    "First Name": "Halsell",
     "Gender": "female",
     "Last Name": "Ann",
     "Link": "http://austintexas.gov/news/traffic-fatality-26-1",
@@ -1854,7 +1856,7 @@
     "Date": "04/29/2017",
     "Ethnicity": "Hispanic",
     "Fatal crashes this year": "24",
-    "First Name": "Kris",
+    "First Name": "Lugo",
     "Gender": "male",
     "Last Name": "Anthony",
     "Link": "http://austintexas.gov/news/traffic-fatality-24-3",
@@ -1869,7 +1871,7 @@
     "Date": "04/27/2017",
     "Ethnicity": "White",
     "Fatal crashes this year": "25",
-    "First Name": "Bobby",
+    "First Name": "Duke",
     "Gender": "male",
     "Last Name": "Joe",
     "Link": "http://austintexas.gov/news/traffic-fatality-25-2",
@rgreinho

This comment has been minimized.

Copy link
Member

commented Apr 22, 2019

There is a code path in def parse_deceased_field which needs to be tested.

It should be when a deceased field is found, when there is a DOB, but then no gender or ethnicity.

@rgreinho

This comment has been minimized.

Copy link
Member

commented Apr 23, 2019

Here is a patch that should help you reach 100% coverage (scrapd/core/apd.py 226 0 100%):

diff --git a/tests/core/test_apd.py b/tests/core/test_apd.py
index ca9d48a..7ef33fa 100644
--- a/tests/core/test_apd.py
+++ b/tests/core/test_apd.py
@@ -220,11 +220,31 @@ def test_extract_traffic_fatalities_page_details_link_00(news_page):
     assert actual == expected
 
 
-def test_parse_deceased_field():
-    field = "Rosbel “Rudy” Tamez, Hispanic male (D.O.B. 10-10-54)".split()
+@pytest.mark.parametrize('input_,expected', (
+    (
+        'Rosbel "Rudy" Tamez, Hispanic male (D.O.B. 10-10-54)',
+        {
+            Fields.DOB: '10-10-54)',
+            Fields.GENDER: 'male',
+            Fields.ETHNICITY: 'Hispanic',
+            Fields.LAST_NAME: 'Tamez',
+            Fields.FIRST_NAME: 'Rosbel',
+        },
+    ),
+    (
+        'DOB: 01-01-99',
+        {
+            Fields.DOB: '01-01-99',
+            Fields.LAST_NAME: '',
+            Fields.FIRST_NAME: '',
+        },
+    ),
+))
+def test_parse_deceased_field(input_, expected):
+    """Ensure the deceased field gets parsed."""
+    field = input_.split()
     d = apd.parse_deceased_field(field)
-    assert d[Fields.LAST_NAME] == "Tamez"
-    assert d[Fields.FIRST_NAME] == "Rosbel"
+    assert d == expected
 
 
 @pytest.mark.parametrize('name,expected', (

mscarey added some commits Apr 23, 2019

move split method within deceased field parser
Allows the parse_deceased_field to be tested by passing in strings, not split lists
split deceased field on some slashes
To handle gender and ethnicity fields in a format like "W/F"

@mergify mergify bot merged commit 3cb9934 into scrapd:master Apr 23, 2019

7 checks passed

Mergify — Summary 1 rule matches
Details
ci/circleci: docs Your tests passed on CircleCI!
Details
ci/circleci: format Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
ci/circleci: prepare Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
coverage/coveralls Coverage increased (+0.09%) to 97.544%
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.