Hart #458

moldover · 2020-07-06T02:10:33Z

Add Hart xml reading for #457 (The Hart CVR structure is defined in rcv/reference/Hart_Verity_CVR_Structure.pdf)
During CVR reading this code filters CVRs by contest name (instead of contest Id).
I had to refactor contest Id into a String (was Integer) which allows it to be used across Dominion and Hart (supports #456).
I added two Hart unit tests which support RCV: hart_cedar_park_school_board and hart_travis_county_officers

tarheel

Nice work! I'll let @HEdingfield take a look also.

src/main/java/network/brightspots/rcv/GuiConfigController.java

src/main/java/network/brightspots/rcv/HartCvrReader.java

src/main/java/network/brightspots/rcv/ResultsWriter.java

src/main/java/network/brightspots/rcv/CastVoteRecord.java

tarheel

@HEdingfield you can take a look!

src/main/java/network/brightspots/rcv/HartCvrReader.java

HEdingfield

Does it make sense to include a PDF in test_data? Seems like it might be better in the reference folder.

src/test/resources/network/brightspots/rcv/test_data/hart/Canvass Results-14-12-2018 19.32.07.pdf

...ghtspots/rcv/test_data/hart_cedar_park_school_board/hart_cedar_park_school_board_config.json

src/main/java/network/brightspots/rcv/CastVoteRecord.java

src/main/java/network/brightspots/rcv/HartCvrReader.java

HEdingfield · 2020-07-08T19:24:20Z

...rightspots/rcv/test_data/hart_travis_county_officers/hart_travis_county_officers_config.json

+    "generateCdfJson" : false
+  },
+  "cvrFileSources" : [ {
+    "filePath" : "CVRArchive_18_9_0_6298/",


This test fails for me with:

2020-07-08 12:20:29 PDT INFO: Reading Hart cast vote record file: 04917dca-3f9e-440e-a41a-98dc5e57376e.xml...
2020-07-08 12:20:29 PDT SEVERE: Error parsing cast vote record:
com.fasterxml.jackson.core.JsonParseException: Unexpected first character (char code 0xEF), not valid in xml document: could be mangled UTF-8 BOM marker. Make sure that the Reader uses correct encoding or pass an InputStream instead
2020-07-08 12:20:29 PDT SEVERE: Error opening cast vote record file: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\CVRArchive_18_9_0_6298
2020-07-08 12:20:29 PDT INFO: Check file path and permissions and make sure they are correct!

Can you post the preceding log output? I'm wondering if it only happens for this file or all of them.

Do these tests not fail for you? Here's the whole thing:

2020-07-09 22:21:11 PDT INFO: Starting tabulation session... 2020-07-09 22:21:11 PDT INFO: Successfully loaded contest config: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\hart_travis_county_officers_config.json 2020-07-09 22:21:11 PDT INFO: Validating contest config... 2020-07-09 22:21:11 PDT INFO: Contest config validation successful. 2020-07-09 22:21:11 PDT INFO: Tabulation logging to: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\output\2020-07-09_22-21-11_audit_*.log 2020-07-09 22:21:11 PDT INFO: Computer name: SMILODON 2020-07-09 22:21:11 PDT INFO: User name: UserName 2020-07-09 22:21:11 PDT INFO: Begin config file contents: 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "tabulatorVersion" : "1.1.0", 2020-07-09 22:21:11 PDT INFO: "outputSettings" : { 2020-07-09 22:21:11 PDT INFO: "contestName" : "TRAVIS COUNTY OFFICERS", 2020-07-09 22:21:11 PDT INFO: "outputDirectory" : "output", 2020-07-09 22:21:11 PDT INFO: "contestDate" : "", 2020-07-09 22:21:11 PDT INFO: "contestJurisdiction" : "", 2020-07-09 22:21:11 PDT INFO: "contestOffice" : "", 2020-07-09 22:21:11 PDT INFO: "tabulateByPrecinct" : false, 2020-07-09 22:21:11 PDT INFO: "generateCdfJson" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: "cvrFileSources" : [ { 2020-07-09 22:21:11 PDT INFO: "filePath" : "CVRArchive_18_9_0_6298/", 2020-07-09 22:21:11 PDT INFO: "firstVoteColumnIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "firstVoteRowIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "idColumnIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "precinctColumnIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "provider" : "HART" 2020-07-09 22:21:11 PDT INFO: } ], 2020-07-09 22:21:11 PDT INFO: "candidates" : [ 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Michael Gottner", 2020-07-09 22:21:11 PDT INFO: "code" : "bdc77efc-2f86-48d0-a0de-2efb02e1f4d9", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Jim Sylvester", 2020-07-09 22:21:11 PDT INFO: "code" : "bfb4eda8-35e8-4d13-b152-99b8e5d6b777", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "William Poole", 2020-07-09 22:21:11 PDT INFO: "code" : "24d59598-4ded-428f-842c-08921c8ed039", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Bruce Elfant", 2020-07-09 22:21:11 PDT INFO: "code" : "fa9ebe45-dd38-423d-a2a8-0bc2de9967f9", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Sheriff Hamilton", 2020-07-09 22:21:11 PDT INFO: "code" : "39a32be4-1804-467c-bf3e-06b174f892d7", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: } 2020-07-09 22:21:11 PDT INFO: ], 2020-07-09 22:21:11 PDT INFO: "rules" : { 2020-07-09 22:21:11 PDT INFO: "tiebreakMode" : "random", 2020-07-09 22:21:11 PDT INFO: "overvoteRule" : "exhaustImmediately", 2020-07-09 22:21:11 PDT INFO: "winnerElectionMode" : "standard", 2020-07-09 22:21:11 PDT INFO: "randomSeed" : "9", 2020-07-09 22:21:11 PDT INFO: "numberOfWinners" : "1", 2020-07-09 22:21:11 PDT INFO: "multiSeatBottomsUpPercentageThreshold" : "", 2020-07-09 22:21:11 PDT INFO: "decimalPlacesForVoteArithmetic" : "4", 2020-07-09 22:21:11 PDT INFO: "minimumVoteThreshold" : "0", 2020-07-09 22:21:11 PDT INFO: "maxSkippedRanksAllowed" : "1", 2020-07-09 22:21:11 PDT INFO: "maxRankingsAllowed" : "max", 2020-07-09 22:21:11 PDT INFO: "nonIntegerWinningThreshold" : false, 2020-07-09 22:21:11 PDT INFO: "hareQuota" : false, 2020-07-09 22:21:11 PDT INFO: "batchElimination" : false, 2020-07-09 22:21:11 PDT INFO: "exhaustOnDuplicateCandidate" : false, 2020-07-09 22:21:11 PDT INFO: "treatBlankAsUndeclaredWriteIn" : false, 2020-07-09 22:21:11 PDT INFO: "overvoteLabel" : "", 2020-07-09 22:21:11 PDT INFO: "undervoteLabel" : "", 2020-07-09 22:21:11 PDT INFO: "undeclaredWriteInLabel" : "", 2020-07-09 22:21:11 PDT INFO: "rulesDescription" : "" 2020-07-09 22:21:11 PDT INFO: } 2020-07-09 22:21:11 PDT INFO: } 2020-07-09 22:21:11 PDT INFO: End config file contents. 2020-07-09 22:21:11 PDT INFO: Tabulating 'TRAVIS COUNTY OFFICERS'... 2020-07-09 22:21:11 PDT INFO: Parsing cast vote records... 2020-07-09 22:21:11 PDT INFO: Reading Hart cast vote records from folder: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\CVRArchive_18_9_0_6298... 2020-07-09 22:21:11 PDT INFO: Reading Hart cast vote record file: 04917dca-3f9e-440e-a41a-98dc5e57376e.xml... 2020-07-09 22:21:11 PDT SEVERE: Error parsing cast vote record: com.fasterxml.jackson.core.JsonParseException: Unexpected first character (char code 0xEF), not valid in xml document: could be mangled UTF-8 BOM marker. Make sure that the Reader uses correct encoding or pass an InputStream instead 2020-07-09 22:21:11 PDT SEVERE: Error opening cast vote record file: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\CVRArchive_18_9_0_6298 2020-07-09 22:21:11 PDT INFO: Check file path and permissions and make sure they are correct! 2020-07-09 22:21:11 PDT SEVERE: Parsing cast vote records failed! 2020-07-09 22:21:11 PDT SEVERE: Aborting tabulation due to cast vote record errors! 2020-07-09 22:21:11 PDT INFO: Tabulation session completed. 2020-07-09 22:21:11 PDT INFO: Successfully loaded contest config: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\hart_travis_county_officers_config.json 2020-07-09 22:21:11 PDT SEVERE: File not found! java.io.FileNotFoundException: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\output\2020-07-09_22-21-11_summary.json (The system cannot find the file specified)

Yeah, they are passing fine for me. I'll see if there is some stray byte here somehow.

@HEdingfield There is indeed a BOM byte order mark on these xml files, which is no problem running on mac, but does seem like a known thing: https://stackoverflow.com/questions/1835430/byte-order-mark-screws-up-file-reading-in-java

I pushed some test code to HartCvrReader:readCastVoteRecord. Can you give that a shot and see if Hart tests complete on Windows? There is another option which is apache io commons has a file stream BOM reader, which we can pull in if needed: https://commons.apache.org/proper/commons-io/javadocs/api-2.2/org/apache/commons/io/input/BOMInputStream.html

The initial fix didn't work, but the other one you left in there with the file stream did! Pushed commit where all my tests are passing now. If they pass on your end too, I think we're good to go!

HEdingfield · 2020-07-08T19:26:26Z

...ghtspots/rcv/test_data/hart_cedar_park_school_board/hart_cedar_park_school_board_config.json

+    "generateCdfJson" : false
+  },
+  "cvrFileSources" : [ {
+    "filePath" : "CVRArchive_18_9_0_6298/",


This test fails for me with:

2020-07-08 12:20:54 PDT INFO: Reading Hart cast vote record file: 04917dca-3f9e-440e-a41a-98dc5e57376e.xml...
2020-07-08 12:20:54 PDT SEVERE: Error parsing cast vote record:
com.fasterxml.jackson.core.JsonParseException: Unexpected first character (char code 0xEF), not valid in xml document: could be mangled UTF-8 BOM marker. Make sure that the Reader uses correct encoding or pass an InputStream instead
2020-07-08 12:20:54 PDT SEVERE: Error opening cast vote record file: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_cedar_park_school_board\CVRArchive_18_9_0_6298
2020-07-08 12:20:54 PDT INFO: Check file path and permissions and make sure they are correct!

src/main/java/network/brightspots/rcv/GuiConfigController.java

HEdingfield · 2020-07-08T19:37:17Z

src/main/java/network/brightspots/rcv/StreamingCvrReader.java

@@ -186,7 +186,11 @@ private void endCvr() {

    if (idColumnIndex != null && currentSuppliedCvrId == null) {
      Logger.log(
-          Level.SEVERE, "Cast vote record identifier not found for: %s", computedCastVoteRecordId);
+          Level.SEVERE,
+          "Cast vote record identifier missing on row %d in file %s.  This may be due to an "


Recommend copying in the below because nits:

"Cast vote record identifier missing in row %d in file:\n%s\n\nThis may be due to an incorrectly formatted *.xlsx file. Try copying your CVR data into a new *.xlsx file to fix this.",

grunted the anti-pedantic Canadian

src/main/java/network/brightspots/rcv/TabulatorSession.java

HEdingfield · 2020-07-10T05:26:34Z

build.gradle

@@ -14,6 +14,7 @@ repositories {
 dependencies {
    compile "org.apache.commons:commons-csv:1.8"
    compile "org.apache.poi:poi-ooxml:4.1.2"
+    compile "com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.10.4"


If you pull my update_deps branch into this PR (or more likely pull develop into here after that branch is approved and merged into it), could you please make sure to update this number to the version that's in there for all the other jackson libraries?

src/main/java/module-info.java

HEdingfield · 2020-07-10T05:28:42Z

src/main/java/network/brightspots/rcv/StreamingCvrReader.java

@@ -186,7 +186,11 @@ private void endCvr() {

    if (idColumnIndex != null && currentSuppliedCvrId == null) {
      Logger.log(
-          Level.SEVERE, "Cast vote record identifier not found for: %s", computedCastVoteRecordId);
+          Level.SEVERE,
+          "Cast vote record identifier missing on row %d in file %s.  This may be due to an "


grunted the anti-pedantic Canadian

HEdingfield · 2020-07-10T05:32:33Z

...rightspots/rcv/test_data/hart_travis_county_officers/hart_travis_county_officers_config.json

+    "generateCdfJson" : false
+  },
+  "cvrFileSources" : [ {
+    "filePath" : "CVRArchive_18_9_0_6298/",


Do these tests not fail for you? Here's the whole thing:

2020-07-09 22:21:11 PDT INFO: Starting tabulation session... 2020-07-09 22:21:11 PDT INFO: Successfully loaded contest config: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\hart_travis_county_officers_config.json 2020-07-09 22:21:11 PDT INFO: Validating contest config... 2020-07-09 22:21:11 PDT INFO: Contest config validation successful. 2020-07-09 22:21:11 PDT INFO: Tabulation logging to: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\output\2020-07-09_22-21-11_audit_*.log 2020-07-09 22:21:11 PDT INFO: Computer name: SMILODON 2020-07-09 22:21:11 PDT INFO: User name: UserName 2020-07-09 22:21:11 PDT INFO: Begin config file contents: 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "tabulatorVersion" : "1.1.0", 2020-07-09 22:21:11 PDT INFO: "outputSettings" : { 2020-07-09 22:21:11 PDT INFO: "contestName" : "TRAVIS COUNTY OFFICERS", 2020-07-09 22:21:11 PDT INFO: "outputDirectory" : "output", 2020-07-09 22:21:11 PDT INFO: "contestDate" : "", 2020-07-09 22:21:11 PDT INFO: "contestJurisdiction" : "", 2020-07-09 22:21:11 PDT INFO: "contestOffice" : "", 2020-07-09 22:21:11 PDT INFO: "tabulateByPrecinct" : false, 2020-07-09 22:21:11 PDT INFO: "generateCdfJson" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: "cvrFileSources" : [ { 2020-07-09 22:21:11 PDT INFO: "filePath" : "CVRArchive_18_9_0_6298/", 2020-07-09 22:21:11 PDT INFO: "firstVoteColumnIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "firstVoteRowIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "idColumnIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "precinctColumnIndex" : "1", 2020-07-09 22:21:11 PDT INFO: "provider" : "HART" 2020-07-09 22:21:11 PDT INFO: } ], 2020-07-09 22:21:11 PDT INFO: "candidates" : [ 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Michael Gottner", 2020-07-09 22:21:11 PDT INFO: "code" : "bdc77efc-2f86-48d0-a0de-2efb02e1f4d9", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Jim Sylvester", 2020-07-09 22:21:11 PDT INFO: "code" : "bfb4eda8-35e8-4d13-b152-99b8e5d6b777", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "William Poole", 2020-07-09 22:21:11 PDT INFO: "code" : "24d59598-4ded-428f-842c-08921c8ed039", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Bruce Elfant", 2020-07-09 22:21:11 PDT INFO: "code" : "fa9ebe45-dd38-423d-a2a8-0bc2de9967f9", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: }, 2020-07-09 22:21:11 PDT INFO: { 2020-07-09 22:21:11 PDT INFO: "name" : "Sheriff Hamilton", 2020-07-09 22:21:11 PDT INFO: "code" : "39a32be4-1804-467c-bf3e-06b174f892d7", 2020-07-09 22:21:11 PDT INFO: "excluded" : false 2020-07-09 22:21:11 PDT INFO: } 2020-07-09 22:21:11 PDT INFO: ], 2020-07-09 22:21:11 PDT INFO: "rules" : { 2020-07-09 22:21:11 PDT INFO: "tiebreakMode" : "random", 2020-07-09 22:21:11 PDT INFO: "overvoteRule" : "exhaustImmediately", 2020-07-09 22:21:11 PDT INFO: "winnerElectionMode" : "standard", 2020-07-09 22:21:11 PDT INFO: "randomSeed" : "9", 2020-07-09 22:21:11 PDT INFO: "numberOfWinners" : "1", 2020-07-09 22:21:11 PDT INFO: "multiSeatBottomsUpPercentageThreshold" : "", 2020-07-09 22:21:11 PDT INFO: "decimalPlacesForVoteArithmetic" : "4", 2020-07-09 22:21:11 PDT INFO: "minimumVoteThreshold" : "0", 2020-07-09 22:21:11 PDT INFO: "maxSkippedRanksAllowed" : "1", 2020-07-09 22:21:11 PDT INFO: "maxRankingsAllowed" : "max", 2020-07-09 22:21:11 PDT INFO: "nonIntegerWinningThreshold" : false, 2020-07-09 22:21:11 PDT INFO: "hareQuota" : false, 2020-07-09 22:21:11 PDT INFO: "batchElimination" : false, 2020-07-09 22:21:11 PDT INFO: "exhaustOnDuplicateCandidate" : false, 2020-07-09 22:21:11 PDT INFO: "treatBlankAsUndeclaredWriteIn" : false, 2020-07-09 22:21:11 PDT INFO: "overvoteLabel" : "", 2020-07-09 22:21:11 PDT INFO: "undervoteLabel" : "", 2020-07-09 22:21:11 PDT INFO: "undeclaredWriteInLabel" : "", 2020-07-09 22:21:11 PDT INFO: "rulesDescription" : "" 2020-07-09 22:21:11 PDT INFO: } 2020-07-09 22:21:11 PDT INFO: } 2020-07-09 22:21:11 PDT INFO: End config file contents. 2020-07-09 22:21:11 PDT INFO: Tabulating 'TRAVIS COUNTY OFFICERS'... 2020-07-09 22:21:11 PDT INFO: Parsing cast vote records... 2020-07-09 22:21:11 PDT INFO: Reading Hart cast vote records from folder: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\CVRArchive_18_9_0_6298... 2020-07-09 22:21:11 PDT INFO: Reading Hart cast vote record file: 04917dca-3f9e-440e-a41a-98dc5e57376e.xml... 2020-07-09 22:21:11 PDT SEVERE: Error parsing cast vote record: com.fasterxml.jackson.core.JsonParseException: Unexpected first character (char code 0xEF), not valid in xml document: could be mangled UTF-8 BOM marker. Make sure that the Reader uses correct encoding or pass an InputStream instead 2020-07-09 22:21:11 PDT SEVERE: Error opening cast vote record file: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\CVRArchive_18_9_0_6298 2020-07-09 22:21:11 PDT INFO: Check file path and permissions and make sure they are correct! 2020-07-09 22:21:11 PDT SEVERE: Parsing cast vote records failed! 2020-07-09 22:21:11 PDT SEVERE: Aborting tabulation due to cast vote record errors! 2020-07-09 22:21:11 PDT INFO: Tabulation session completed. 2020-07-09 22:21:11 PDT INFO: Successfully loaded contest config: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\hart_travis_county_officers_config.json 2020-07-09 22:21:11 PDT SEVERE: File not found! java.io.FileNotFoundException: C:\Users\UserName\IdeaProjects\rcv\src\test\resources\network\brightspots\rcv\test_data\hart_travis_county_officers\output\2020-07-09_22-21-11_summary.json (The system cannot find the file specified)

moldover · 2020-07-11T02:49:35Z

Read and tabulate Hart CVRs #457 #458

* add contest id to config * add xml parsing module * First cut at Hart CVR reader + test assets. * cleanup paths and add expected results * add full cvr data * refactor contestId from Integer to String * add school board test * fix xml parsing * logging * re-add accidentally removed import * cleanup for PR#548 * updates for PR #458 * updates for PR #458 * test other file reading options * more cleanup for PR #458 * more cleanup for PR #458 * Fixes tests not working in Windows due to mangled UTF-8 BOM marker. * Handles potential NPE if chosen directory contains no files. Co-authored-by: HEdingfield <hylton@groupagree.com>

@chughes297

* Changed package from com.rcv to network.brightspots.rcv. * remove unused tests: fix for #197 * add tiebreak test #197 * Allows user to cancel during interactive tiebreaker (fixes #132); adds example config file for testing interactive tiebreaker with sequential multi-seat enabled; adds commented command in build.gradle for executing CLI arguments. * add missing precinct test * Reconciles branch with `develop`. * Re-adds interactive tiebreaker sample input. * Continues fixing recently broken tests. * Reverts name back to "Universal RCV Tabulator". * add new random seed config field * updated tests * incorporate new option into tabulation logic and add test case * onlyOneWinnerPerRound -> allowOnlyOneWinnerPerRound * update test case to include a tie-break * update tests * address comments * tests * add test case * update one test for each winnerElectionMode option * updating tests * fix tests * fix minor output bug and add test case for previousRoundCountsThenRandom tie-break option * fix minor issue with transfer tallies * update test configs * update a ton of JSON files! * fix bad source path * Increment version from 0.1.0 to 0.1.1. (#373) * Increment version from 0.1.1 to 0.1.2 * Increment version from 0.1.2 to 1.0.0 (#384) * v1.0.0 * Increment version from 1.0.0 to 1.0.1 (#389) * Fix broken tests (#393) * Increments version from "1.0.1" to "1.0.1_20200320_internal" (#423) * Adds multiSeatBottomsUpPercentageThreshold (#428) * Tests for Dominion CSV conversion routine (#429) * test for Dominion csv conversion routine #415 * 2 more tests for cvr conversion * updates for Dominion conversion tests: PR #425 * fix for PR #425 Co-authored-by: jonmoldover <jonmoldover@gmail.com> * Adds AUTOMATED_TEST_VERSION code so tests can remain stable as app version updates (fixes #426). (#435) * Fixes Dominion conversion output filenames and path (#436) * Dominion conversion output files now output to same directory as input files and include timestamp in filename (#408). * Updates Dominion tests to account for new filename schema; Dominion test files are now cleaned up automatically; fixes bug with fileCompare not closing readers. * All delete operations in `TabulatorTests` now handle exceptions with useful error messages if they fail. * read and write precinct portion and precinct (#433) * read and write precinct portion + precinct * edits for precinct portion PR#433 * Update src/main/java/network/brightspots/rcv/DominionCvrReader.java Co-Authored-By: HEdingfield <HEdingfield@users.noreply.github.com> * Update src/main/java/network/brightspots/rcv/DominionCvrReader.java Co-Authored-By: HEdingfield <HEdingfield@users.noreply.github.com> Co-authored-by: HEdingfield <HEdingfield@users.noreply.github.com> * UWI test (#440) * add UWI test files * add an additional test to validate my fix of #397 * use a better name for the first test * make batch elimination work properly with singleSeatContinueUntilTwoCandidatesRemain (#441) * make batch elimination work properly with singleSeatContinueUntilTwoCandidatesRemain * cleaned up wording * v1.1.0 (#454) ### New features: * Added support for converting Dominion JSON CVRs to generic .csv format (including precinct portions) (#404, #406, #407, #408, #415, #439) * Added multiSeatBottomsUpPercentageThreshold option (#403) * Added CLI option to convert Dominion CVR to generic .csv (#408) * New GUI menu and conversion options (can now convert to CDF and convert Dominion to generic via the GUI) (#408, #421) * Added Dominion Alaska CVR to sample_input folder ### Bug fixes: * Batch elimination now works properly with singleSeatContinueUntilTwoCandidatesRemain (#396) * In a multi-seat contest, if someone wins in the first round, we now automatically eliminate undeclared write-ins before we eliminate any other candidates; previously, we treated UWIs like a normal candidate, which meant we potentially eliminated other candidates with lower tallies first (#397) * If UWI exceeds the winning threshold in the initial count, we no longer mistakenly elect this candidate (#398) ### Backend updates: * Updated dependencies to latest version: JDK, JavaFX, Checkstyle google_checks.xml, Checkstyle plugin, org.openjfx.javafxplugin, org.beryx.jlink, org.apache.commons:commons-csv, org.apache.poi:poi-ooxml, com.fasterxml.jackson.core:jackson-* * Added special code to test configs to obviate the need to update the version with each increment (#426) * Updated tests and improved test coverage * Copyright update (#414) * Code cleanup * Hart (#458) * add contest id to config * add xml parsing module * First cut at Hart CVR reader + test assets. * cleanup paths and add expected results * add full cvr data * refactor contestId from Integer to String * add school board test * fix xml parsing * logging * re-add accidentally removed import * cleanup for PR#548 * updates for PR #458 * updates for PR #458 * test other file reading options * more cleanup for PR #458 * more cleanup for PR #458 * Fixes tests not working in Windows due to mangled UTF-8 BOM marker. * Handles potential NPE if chosen directory contains no files. Co-authored-by: HEdingfield <hylton@groupagree.com> * Update dependencies (#459) * Updates to latest version: * Gradle 6.5.1 * JDK 14.0.1 * JavaFX 14.0.1 * Checkstyle google_checks.xml 8.34 * Checkstyle plugin 8.34 * org.openjfx.javafxplugin 0.0.9 * org.beryx.jlink 2.20.0 * com.fasterxml.jackson.core:jackson-* 2.11.1 * org.junit.jupiter.junit-jupiter-* 5.6.2 * Gets tests working with Gradle 6.5.1 (fixes #283) * Updates `APP_VERSION` to temporary value 1.2.0_alpha * Addresses Checkstyle warnings * Fixes versions in Hart tests * make contestId a real config field (#463) * make contestId a real config field * update .txt * Providers dropdown (#462) * Changes "Provider" field for CVR source to be required, bases it on an enum of valid values, and changes it to a ChoiceBox at the start of the "CVR Files" tab in the GUI (progress on #460 and #461). * Adds log message to help identify which CVR source validation errors pertain to. * Gets rid of `SimpleObjectProperty` warnings during compile (fixes #280). * Hart GUI (#466) * Provider ComboBox now dynamically enables and disables fields and buttons when different providers are selected. (Progress on #461... actually progress on GUI phase 2!) * Enables `DirectoryChooser` for Hart so it now functions through the GUI. (Fixes #460) * Validation improved to fail if `firstVoteColumnIndex`, `firstVoteRowIndex`, `idColumnIndex`, `precinctColumnIndex` are defined for CVRs from non-ES&S providers. * Removes superfluous logging. * direct Dominion tabulation (#470) * direct Dominion tabulation * enable specifying a Dominion CVR source path in the UI * output generic CSV and add test coverage * remove extra log line * filter out CVRs from other contests during the loading * Changes `contestId` so it's specified on a per-CVR basis (fixes #456) (#474) * First cut at Clear Ballot rcv reader (#475) * First cut at Clear Ballot rcv reader Add Kansas Dem Primary regression test Add Clear Ballot reference files #400 * updates for PR #475 * updates for PR #475 * updates for PR #475 * Enable Clear Ballot tabulation in GUI (progress on #400) (#480) * GUI redesign phase 1 (#492) * Splits Output tab into new Contest Info and Output tabs. * Redesigns GUI CVR Files tab, adds Clear button, and changes Add button so it only clears the file path to make it easier to manually enter multiple sources that share fields. * Improves visual presentation of Candidate tab; adds Clear button and adds checkBoxCandidateExcluded when adding a candidate. * Reorganizes presentation of rules in "Winning Rules" and "Voter Error Rules" tabs. * Winner Election Mode and Tiebreak Mode now start undefined with all relevant fields disabled; choosing specific modes enables applicable fields. * Changes Winner Election Modes and Tiebreak Modes to be more user-friendly, including necessary migration logic to update older config files. * Adds `continueUntilTwoCandidatesRemain` boolean (fixes #481). * Expands footprint of GUI window to 1200x1000. * Implements bordered boxes. * remove Dominion conversion feature (#487) * update test files * remove conversion feature from code * update test configs * remove unnecessary check * allow overvote delimiter for ES&S source files (#503) * test * update winner election mode logic * comments * final comments * Read and Tabulate Unisyn Xml Cdf (#504) * plumbing for xml cdf reading * add contestId to json and xml CDF parse logic update unisyn cvr test to use the full (but still incorrect) cvr data. This test is expected to fail. add candidate parsing code to xml CDF parser * dont parse Candidate data from CDF at runtime - this enforces the same constraints on validation as all other file types. The un-used candidate parsing code is left in for implementing future helper functions which will automatically populate config files. * Handle ContestSelections in XML correctly * add NIST example 2 (seems to be correctly formed) * fill out CDF XML class definitions needed for tabulation * add some helpers * cleanup pre-processing code * add more class fields around contest selections fix ranking parsing add example Unisyn contests * handle write-ins * get first 2 Unisyn regression tests working :) * fix first NIST CDF XML test * cleanup test names * parse GpUnit from CVRs * remove un-used assets * updates for PR #504 * updated data from Unisyn with GpUnit (precinct) parsing validated * add more Unisyn regression tests tests share a single cvr input file * tabulate all elections - in practice we should not see more than one * updates for PR #504 * updates for PR #504 * Sundry UX fixes (#509) * Converts `overvoteRule` from a `ChoiceBox` to an array of `RadioButton`s; changes `overvoteRule` string display in config files and adds migration logic. * Disables `decimalPlacesForVoteArithmetic` and `nonIntegerWinningThreshold` except when `winnerElectionMode` is "Multi-winner allow only one winner per round" or "Multi-winner allow multiple winners per round" (fixes #500). * Adds suggested values for `overvoteLabel`, `undervoteLabel`, and ES&S column and row indices as requested by @chughes297 (overriding what was in the original GUI mock-ups). * Clarifies language to address confusing UX in `passesBasicCvrSourceValidation()` for CVR-specific errors. * Cdf Json Fixes (#506) * plumbing for xml cdf reading * add contestId to json and xml CDF parse logic update unisyn cvr test to use the full (but still incorrect) cvr data. This test is expected to fail. add candidate parsing code to xml CDF parser * dont parse Candidate data from CDF at runtime - this enforces the same constraints on validation as all other file types. The un-used candidate parsing code is left in for implementing future helper functions which will automatically populate config files. * Handle ContestSelections in XML correctly * add NIST example 2 (seems to be correctly formed) * fill out CDF XML class definitions needed for tabulation * add some helpers * cleanup pre-processing code * add more class fields around contest selections fix ranking parsing add example Unisyn contests * handle write-ins * get first 2 Unisyn regression tests working :) * fix first NIST CDF XML test * cleanup test names * parse GpUnit from CVRs * remove un-used assets * updates for PR #504 * updated data from Unisyn with GpUnit (precinct) parsing validated * add more Unisyn regression tests tests share a single cvr input file * tabulate all elections - in practice we should not see more than one * updates for PR #504 * better logging * first cut at fixing JSON CDF reader: - all static election objects are parsed at beginning of cvr processing - during processing contest selections are linked to candidate objects - candidates are stored in config - not pulled from CDF file - fixed NIST test set 0 config and CDF json to match the NIST 1500-103 UML - verified tabulation results are correct for Nist test set 0 test * update ResultsWriter CVR generation code to create Candidate objects and link to them from ContestSelection objects. verified updated NIST test 0 cvr output * updates for PR #504 * fix typo * update test asset with contest name * better handling for cdf reader parse errors * update all CDF json assets * Updates to CDF parsing logic: Candidate validation and CandidateId validation are done while parsing CVRs (not during static parsing) xml parser throws unrecognizedCandidatesException (instead of just CVRParserException) Add json getCvrContest helper json parser logic upgrades: handle writeIns in both places check for rank in both places check for HasIndication check for IsAllocable synthesize cvrId * cleanup for PR #506 * updates for PR #505 * Throw if undeclared write-in is found but label has not been defined. * make some fields per-source instead of per-config (#517) * create new class * helpful errors in CLI; don't allow newer version in config than in app * make fields per-source * comments * Jon's changes * fix almost all tests * fix Dominion tests * update validation and documentation for 4 fields that moved to CVR level (#518) * update validation and documentation for 4 fields that moved to CVR level * Fixes broken tests and a couple of typos. Co-authored-by: HEdingfield <hylton@groupagree.com> * camelCase strings for enums in config files (#519) * use camelCase for enum values in configs and update migration logic accordingly * disable editing of CVR source rows in GUI table * Final 1.2.0 polish (#526) * Updates checkstyle from 8.34 to 8.36.2, and google_checks.xml. * Creates and implements separate `checkstyle-suppressions.xml` file to suppress Checkstyle warnings that don't make sense for our project (fixes #489). * Addresses relevant Checkstyle warnings (fixes #490). * Addresses IntelliJ warnings. * Pixel-pushing for Linux (get thicc, boi) * Changes all logging to use new `fine()`, `info()`, `warning()`, and `severe()` methods. * Gets rid of now-unused `onEditCommit` functions for the CVR Files and Candidates tables. * Makes util classes `final`, with private constructors to prevent instantiation. * Renames `TieBreakMode`, `tieBreakMode`, `TieBreak`, and `tieBreak` to `TiebreakMode`, `tiebreakMode`, `Tiebreak`, and `tiebreak`, respectively. This word has now lost all meaning. * Removes unnecessary usages of `.toString()`. * Standardizes exceptions as `exception` instead of `e` to comply with VVSG requirement on variable names (5.2.5 paragraph c) that only index loops are allowed to have single-character variable names. * Fixes broken test. * Minor fixes to hints. * Get rid of the Devilish Double Spaces after periods. * Minor cleanups (#527) * fix README * CDF JSON: throw if a CandidateObject can't be found during parsing. * Cleanup CDF class (type) names as output from ResultsWriter. * fix label * Handle missing precinct data for older Dominion data sets #533 (#534) * Handle missing precinct data for older Dominion data sets #533 * Add test data with missing precinct data for older Dominion data sets #533 * Fix for #536 - allow multiple CDF files (#537) * Fix for #536 - allow multiple CDF files Add regression test with multiple CDF files * fix overvote delimiter logic and update test (#539) * Fixes for bad merge between `master` and `develop`. * Add README and LICENSE for split repo * latest updates from develop * update cdf output names for #584 * add dominion multi file tests * add licence file #551 * remove test_data as submodule #612 * remove test_data submodule #612 * remove submodule reference from docs #612 Co-authored-by: HEdingfield <hylton+git@groupagree.com> Co-authored-by: hedingfield <hylton@groupagree.com> Co-authored-by: Louis Eisenberg <tarheel@gmail.com> Co-authored-by: HEdingfield <HEdingfield@users.noreply.github.com> Co-authored-by: Armin Samii <artoonie@gmail.com> Co-authored-by: Armin Samii <armin.samii@gmail.com>

moldover added 9 commits June 14, 2020 13:28

add contest id to config

ad2eb4f

add xml parsing module

c072f78

First cut at Hart CVR reader + test assets.

7f845d0

cleanup paths and add expected results

9b91965

add full cvr data

6f55dcc

refactor contestId from Integer to String

f52ce58

add school board test

77a5293

fix xml parsing

07e1658

logging

d2f0604

moldover requested review from tarheel and HEdingfield July 6, 2020 02:10

re-add accidentally removed import

5c6f774

tarheel reviewed Jul 6, 2020

View reviewed changes

tarheel reviewed Jul 7, 2020

View reviewed changes

src/main/java/network/brightspots/rcv/CastVoteRecord.java Show resolved Hide resolved

cleanup for PR#548

1f8dcf7

tarheel reviewed Jul 8, 2020

View reviewed changes

src/main/java/network/brightspots/rcv/HartCvrReader.java Outdated Show resolved Hide resolved

src/main/java/network/brightspots/rcv/HartCvrReader.java Show resolved Hide resolved

HEdingfield requested changes Jul 8, 2020

View reviewed changes

moldover added 2 commits July 10, 2020 00:24

updates for PR #458

22c0bff

updates for PR #458

372b907

HEdingfield reviewed Jul 10, 2020

View reviewed changes

moldover and others added 5 commits July 10, 2020 10:45

test other file reading options

d2cd7d2

more cleanup for PR #458

ef5ebae

more cleanup for PR #458

6ca889f

Fixes tests not working in Windows due to mangled UTF-8 BOM marker.

ee8b819

Handles potential NPE if chosen directory contains no files.

6ba4c64

HEdingfield approved these changes Jul 11, 2020

View reviewed changes

moldover merged commit 4baba14 into develop Jul 11, 2020

moldover deleted the hart branch July 11, 2020 02:49

HEdingfield mentioned this pull request Jul 11, 2020

Read and tabulate Hart test data #457

Closed

tarheel mentioned this pull request Jul 14, 2020

make contestId a real config field #463

Merged

moldover mentioned this pull request Sep 24, 2020

support Hart format #401

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hart #458

Hart #458

moldover commented Jul 6, 2020 •

edited

Loading

tarheel left a comment

tarheel left a comment

HEdingfield left a comment

HEdingfield Jul 8, 2020

moldover Jul 10, 2020

HEdingfield Jul 10, 2020

moldover Jul 10, 2020

moldover Jul 10, 2020

HEdingfield Jul 10, 2020

HEdingfield Jul 8, 2020

HEdingfield Jul 8, 2020

moldover Jul 10, 2020

HEdingfield Jul 10, 2020

HEdingfield Jul 10, 2020

HEdingfield Jul 10, 2020

HEdingfield Jul 10, 2020

moldover commented Jul 11, 2020

Hart #458

Hart #458

Conversation

moldover commented Jul 6, 2020 • edited Loading

tarheel left a comment

Choose a reason for hiding this comment

tarheel left a comment

Choose a reason for hiding this comment

HEdingfield left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moldover commented Jul 11, 2020

moldover commented Jul 6, 2020 •

edited

Loading