New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLI interface #18

Open
ghing opened this Issue Mar 6, 2017 · 12 comments

Comments

Projects
None yet
3 participants
@ghing
Contributor

ghing commented Mar 6, 2017

While the original use case didn't want to consider how to download and unpack the XML results, I think it will make it a lot easier for volunteers to deal with counties or states that use Clarity systems for their results if they could just run a command to download the results as CSV and then use simpler scripts or manually update the data to do any post-processing.

@chagan and I started working on this at the #NICAR17 hackathon.

Tasks

  • Figure out which values from the Clarity detail XML to include in CSV and how to present them (e.g. # of precincts reporting)
  • Test! Test! Test! At the very least we should have an integration test for both a state and county XML file.
  • Write utility function for parsing jursidiction levels from Clarity URLS (@chagan)
  • Write utility function for retrieving lowest jurisdiction level from Clarity URL (@chagan)
  • Figure out why there are rows with no vote counts when calling clarify results (@ghing)
  • Update CLI to pull levels from url
@ghing

This comment has been minimized.

Contributor

ghing commented Mar 6, 2017

@dwillis, I made this ticket to have a place to track the work that Chris and I have done so far and to reference once I send a pull request.

Should the output columns match, as much as possible, the ones listed in Common Fields? Or the columns listed in the data entry instructions?

Another option that Chris and I discussed would to be to have the output match that from elex so users of that tool could easily pull results from Clarity into a system designed around using elex to pull in AP results. However, our thoughts at the time were that this might be a later feature to add, if someone requests it, and instead prioritize whatever's easiest for Open Elections.

@dwillis

This comment has been minimized.

Contributor

dwillis commented Mar 6, 2017

@ghing my thoughts here are that the output should probably be closer to the columns in the data entry instructions, with the caveat that if there are additional columns they should be provided as well. I like the elex integration idea, but agree that it's a future thing.

@ghing

This comment has been minimized.

Contributor

ghing commented Mar 8, 2017

Note to self: Originally @chagan and I wanted to keep this as simple as possible and envisioned having one command to rule them all that would download all results for all jurisdictions. However, this is both slow and difficult to integration test (because I'm finding only one jurisdiction has issues while the others are fine). I also wonder if grabbing anything is the most common use case. On election night, folks probably just want racewide and for people doing further analysis, running a command for each county, or scripting that separately isn't too onerous. I'm going to try making separate jurisdictions and results subcommands. Having subcommands seems worth it anyway to make things more extensible.

@ghing

This comment has been minimized.

Contributor

ghing commented Apr 10, 2017

Just an update on this. I wrote almost all of the code to do this at the beginning of the month, but in testing for the use case, I ran into some errors for some of the counties. I'm slammed with other projects, at least for this week, but I'm hoping to circle back around to this at the end of the month or beginning of May.

ghing added a commit to ghing/clarify that referenced this issue Apr 20, 2017

Add CLI interface
Start adding a CLI interface to this package that will fetch a list of
jurisdictions from a parent jurisdiction and download and parse results
as CSV.

This is a checkpoint commit so I can share the code with @chagan.  It
still needs a lot of work.

Addresses openelections#18
openelections#18
@ghing

This comment has been minimized.

Contributor

ghing commented Apr 20, 2017

A little background on the empty results:

clarify results http://results.enr.clarityelections.com/AR/Yell/63988/184121/Web01/en/summary.html

produces these results:

state,jurisdiction,office,candidate,party,votes
,,U.S. President & Vice President,,,
,Bluffton,U.S. President & Vice President,,,
,Briggsville,U.S. President & Vice President,,,
,Centerville,U.S. President & Vice President,,,
,Compton,U.S. President & Vice President,,,
,Crawford,U.S. President & Vice President,,,
,Danville,U.S. President & Vice President,,,
,Dardanelle Outside,U.S. President & Vice President,,,
,Dardanelle Ward 1,U.S. President & Vice President,,,
,Dardanelle Ward 2,U.S. President & Vice President,,,
,Dardanelle Ward 3 Other,U.S. President & Vice President,,,
,Dardanelle Ward 3 JP 10,U.S. President & Vice President,,,
,Dutch Creek,U.S. President & Vice President,,,
,Ferguson,U.S. President & Vice President,,,
,Galla Rock,U.S. President & Vice President,,,
,Gilkey,U.S. President & Vice President,,,
,Gravelly,U.S. President & Vice President,,,
,Herring,U.S. President & Vice President,,,
,Lamar,U.S. President & Vice President,,,
,Magazine 1,U.S. President & Vice President,,,
,Magazine 2,U.S. President & Vice President,,,
,Mason,U.S. President & Vice President,,,
,Mountain,U.S. President & Vice President,,,
,Prairie,U.S. President & Vice President,,,
,Richland,U.S. President & Vice President,,,
,Waveland,U.S. President & Vice President,,,
,Riley,U.S. President & Vice President,,,
,Rover,U.S. President & Vice President,,,
,Ward,U.S. President & Vice President,,,
,Ions Creek,U.S. President & Vice President,,,

I need to figure out why there are result objects that don't have any votes.

@ghing

This comment has been minimized.

Contributor

ghing commented Jul 2, 2017

I added some logging to the Parser class and realized that my code in clarify.cli.results.result_as_dict was pulling the votes from Choice.total_votes which will be the total votes for the candidate (choice) for a particular type of vote and not the per-precinct votes.

Here's the code that doesn't work:

def result_as_dict(result, **addl_cols):
    """Return a result as a dictionary suitable for serialization"""
    result_dict = dict(**addl_cols)
    result_dict['office'] = result.contest.text
    # Cols:
    #  county, precinct, office, district, party, candidate, votes, winner (if
    #  it's in the data).

    if result.jurisdiction is not None:
        result_dict['jurisdiction'] = result.jurisdiction.name

    if result.choice is not None:
        result_dict['candidate'] = result.choice.text
        result_dict['party'] = result.choice.party
        result_dict['votes'] = result.choice.total_votes

    return result_dict

I'm going to get started on reworking this.

ghing added a commit to ghing/clarify that referenced this issue Jul 2, 2017

Get jurisdiction level from result URL
The data doesn't contain information about the jurisdiction for
the results, that is whether the URL represents state results or
county results.  However, we've found that result URLs are either
for a state or a county and that they have a distinct format.

Based on this, we can detect level from the URL so the user doesn't have
to specify it as an explicit command line argument.

Add a utility function to detect jurisdiction levels from a results URL
and use this function in the CLI commands.

Addresses openelections#18
openelections#18

ghing added a commit to ghing/clarify that referenced this issue Jul 2, 2017

Document the CLI
Add preliminary documentation for the CLI in the README.  This is pretty
sparse.  It's mostly helpful for me to remember the CLI interface as I'm
working on this project in spurts.

Addresses openelections#18
openelections#18

ghing added a commit to ghing/clarify that referenced this issue Jul 2, 2017

Fix votes shown in output CSV
The logic that prepared a `Result` object for CSV output was incorrectly setting the votes to the total votes for each cadidate instead of the votes for each candidate in each jurisdiction.

Include the vote type in the CSV results.

Reorder CSV columns to match examples in http://docs.openelections.net/data-entry/

Add logging in the `Parser` class and an option in the `results` command to make it easier to identify problems with parsing or application logic.

Addresses openelections#18
openelections#18
@ghing

This comment has been minimized.

Contributor

ghing commented Jul 2, 2017

While figuring out the problem I reported in this comment, I realized that there are a bunch of fields in the XML that we could surface in the CSV.

Jurisdiction:

  • total voters
  • ballots cast
  • voter turnout
  • percent reporting
  • precincts participating
  • precincts reported
  • precincts reporting percent

Contest:

  • # of candidates you can vote for
  • is the contest a ballot question
  • percent reporting
  • precincts participating
  • precincts reported
  • # of counties reporting

We should figure out which of these should be included in the output CSV and how they should be presented.

My intuition is that we should:

  • Include everything
  • Repeat values for each result that's for a given contest, jurisdiction
  • Leave values recorded at the county level (e.g. precincts reporting) blank for precinct results

@dwillis, do you have thoughts or questions?

@dwillis

This comment has been minimized.

Contributor

dwillis commented Jul 2, 2017

@ghing

This comment has been minimized.

Contributor

ghing commented Sep 6, 2018

@GPHemsley I am so excited to see you giving some love to this project. I have some commits tucked away from when I was working on adding a CLI to this project. I wanted to check in and see if this was something you were also poking at. I'll try to merge in your improvements and just take stock of where I was at with adding a CLI.

@GPHemsley

This comment has been minimized.

Collaborator

GPHemsley commented Sep 7, 2018

@ghing I was working my way towards this ticket, per @dwillis, but I don't have to if you have a WIP.

@ghing

This comment has been minimized.

Contributor

ghing commented Sep 7, 2018

@ghing

This comment has been minimized.

Contributor

ghing commented Sep 9, 2018

@GPHemsley I merged master into my feature branch over at https://github.com/ghing/clarify/tree/add-cli. Feel free to use that as a starting point for further work on this feature, and let me know if you have any questions. I'm happy to pass this off to you, but didn't want you to replicate effort.

@GPHemsley GPHemsley added this to the 0.5.0 milestone Sep 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment