Skip to content
Permalink
Branch: dev
Commits on Oct 2, 2019
  1. Upgrade grunt-postcss npm dependency

    stouffers committed Oct 2, 2019
    Upgrade the grunt-postcss npm dependency because one of its
    dependencies, diff, had some security issues.
Commits on Sep 25, 2019
  1. Upgrade npm packages

    stouffers committed Sep 25, 2019
  2. Update maven dependencies

    stouffers committed Sep 25, 2019
  3. Address unchecked warnings

    stouffers committed Sep 25, 2019
    Address some unchecked warnings by adding proper generic declarations or
    suppressing unchecked warnings if unchecked casting is expected and
    unavoidable.
  4. Upgrade apache commons libraries.

    stouffers committed Sep 25, 2019
    Add commons-text, the new home for text based functions from
    commons-lang.  Update/replace usages of deprecated methods.
Commits on Sep 23, 2019
  1. Remove uni-bill text sync for xml processing.

    stouffers committed Sep 23, 2019
    Unibill text syncing is not necessary when processing xml data because
    billtext files may now list multiple bills.  The text syncing has also
    been found to cause some problems.
    
    Fixes #12600
Commits on Sep 9, 2019
  1. Merge pull request #32 from nysenate/api-user-contact-management

    stouffers committed Sep 9, 2019
    Api user contact management
Commits on Aug 28, 2019
  1. Add Spotcheck for NYSenate.gov law data

    stouffers committed Aug 28, 2019
    Adds a spotcheck report that checks nysenate.gov law data.  The data is
    parsed from a data dump in a similar format to other nysenate.gov dumps.
    The law data is checked on several different levels to cover the
    different fields in the data dump:
     - Law Tree based on parent id
     - Law document data
     - Law document relational data
     - List of all law chapters
    
    The front end is updated to accommodate the new law spotcheck data.
    
    Fixes #11998
Commits on Aug 27, 2019
  1. Update spotcheck frontend for new content types

    stouffers committed Aug 27, 2019
    Update the spotcheck report page to account for new content types and
    differing sets of content types between data sources.
Commits on Aug 26, 2019
  1. Alter mismatch save logic for very large reports

    stouffers committed Aug 26, 2019
    The spotcheck dao was producing errors when saving large reports (>
    ~32000 observations, a little less than max short value) due to a
    limitation of the number of query parameters that can be sent for a
    single query.
    
    This commit refactors the report save process so that observations are
    saved in batches to avoid exceeding the above mentioned limit.  Also,
    some simplification of MismatchUtils.
Commits on Aug 20, 2019
  1. Refactor spotcheck dao class structure

    stouffers committed Aug 20, 2019
    Significantly modify the spotcheck dao class structure such that there
    is only one spotcheck dao instead of multiple subclasses covering each
    content type.
    
    - Remove content type specific daos
    - Modify AbstractSpotCheckReportDao to be able to handle all spotcheck
      functions.  Rename to SqlSpotCheckReportDao
    - Instead of providing key<->hstore mapping functionalities via
      subclassing, do it via a specialized mapper interface with
      implementations for each content type.  The correct mapper is chosen
      using the SpotCheckContentType enum.
    - Add additional SpotCheckContentType values so that each enum maps to
      a single key class.  Assign these new types to the spotcheck ref types
      that are applicable.
    - Modify existing mismatches to incorporate new content types.
  2. Fix issues with cached member service

    stouffers committed Aug 20, 2019
    Fix issue with cached member service where the entire FullMember cache
    is warmed on cache miss.
    
    Also do some restructuring of the MemberService and indivdual member
    cache classes so that there is better encapsulation of cache
    functionality.
    
    Refs #12323
Commits on Aug 15, 2019
  1. Upgrade elasticsearch client to version 7.

    stouffers committed Apr 23, 2019
    Upgrades the version of the elasticsearch client to 7, making
    modifications for breaking changes and newly deprecated features:
    
    - Set es client version in pom to 7.
    - Remove unnecessary dependency on ES server.
    - Implement strict LocalDateTime serialization for elasticsearch to
      address now stricter date mappings.
    - Remove deprecated references to "types" within indices.
    - Switch to non-deprecated versions of index management classes.
    - Update result count code to handle new return type for count.
    
    Fixes #12668
  2. Group common NYSenate.gov reporting functions

    stouffers committed Apr 9, 2019
    Group common NYSenate.gov reporting functions in a base class that is
    now extended by the NYSenate.gov report services.
  3. Move spotcheck util methods out of abstract class

    stouffers committed Apr 9, 2019
    Changes the BaseSpotCheckService class into the SpotCheckUtils class,
    which presents the utility methods of the former as public.  Spotcheck
    services no longer extend the base class.  They now just implement the
    SpotCheckService interface and use the SpotCheckUtils class to provide
    the methods previously inhereted from the base class.
    
    This was done to allow more flexible use of these utility methods.  None
    of the methods depend on the spotcheck service implementation so they
    don't need to be coupled to it.
  4. Fix sponsor parsing issue.

    stouffers committed Apr 2, 2019
    The sponsor parser was misbehaving by interpreting the sponsor line
    "RULES COM" as if there was a sponsor named "COM".  The data here is
    questionable.  Usually the line will be "RULES COM {sponsor name}" and
    will be just "RULES" if there is no associated member sponsor.
    
    This fix handles this case by modifying the rules sponsor regex and
    logic.  There was some unnecessary logic in the parser and some flat out
    bad logic as well.  I have also modified the SessionMember shortname
    regex to handle cases where the first initial has additional characters
    (e.g. MILLER ML).
  5. Fix LDSpon amendment transfer test.

    stouffers committed Apr 2, 2019
    There was an error a while ago in the amendment transfer test in the
    LDSpon integration test suite related to a bill print number that was
    more than 5 digits.  Instead of being fixed this test was commented out.
    
    This commit fixes the amendment transfer test by changing the print no
    used in test logic/data to a valid print no.
Commits on Jul 3, 2019
  1. Fix bug when updating unconsolidated law trees

    stouffers committed Jul 3, 2019
    There was a bug that occurred when updating the law tree for certain
    unconsolidated laws.  This subset of unconsolidated laws do not have
    explicitly defined root documents, so the parser creates a dummy
    document to serve as the root when the law tree is first constructed.
    The problem arises when the law tree is rebuilt for these laws.  When
    this happens, a new dummy tree node is created but not a new dummy
    document.  This causes a foreign key violation when inserting the new
    dummy tree node as it does not reference a valid document.
    
    I have fixed this problem by adding logic to reuse previous dummy roots
    when rebuilding the law tree:
    
    - Updated law document model in code and db to include a dummy flag to
      identify dummy docs.
    - When rebuilding law trees that do not have an explicit root, the
      parser will now reuse the previous root document if it has the dummy
      flag instead of creating a new dummy root.
Commits on Jun 3, 2019
  1. Add tomcat permissions to setup instructions

    stouffers committed Jun 3, 2019
    Add instructions on setting permissions for the tomcat directory to the
    setup instructions.
Commits on May 29, 2019
  1. Add session year parameter validation.

    stouffers committed May 29, 2019
    Adds a validation method that can be used to validate session year
    parameters.  Use this method in several places where session years are
    parsed in the API.
    
    Refs #12698
Commits on May 6, 2019
  1. Modify pubhearing parser for new datetime format.

    stouffers committed May 6, 2019
    Recently, we have started receiving hearings with a slightly different
    date/time format that breaks the current parser.  The dates in the new
    format contain labels ("Date:" and "Time:") for the dates/times.
    
    This commit adjusts the parser to be able to deal with these labels.
    Also, a new test is added to test the parsing of the new date/time
    formats.
    
    Fixes #12704
Commits on May 2, 2019
  1. Fix bug allowing creation of negative session year

    stouffers committed May 2, 2019
    There was a bug in the SessionYear class that allowed for the
    instantiation of a negative session year, even though the creation of a
    negative session is explicitly forbidden.  When a year of 0 is provided,
    it passes the non-negative validation and is then computed to a session
    year of -1.  This has caused issues where a session year of -1 is saved,
    and then causes an exception when it is loaded.
    
    This is addressed in this commit by validating the computed session year
    instead of the directly passed in session year.  Now, when 0 is passed
    in, it will validate the computed value of -1 and throw an exception,
    preventing negative sessions from being persisted with the processors.
    
    Refs #12698
Commits on Mar 11, 2019
  1. Refactor notification subscription model

    stouffers committed Mar 11, 2019
    Some major changes to the notification subscription data model to allow
    for improved and seamless notification digest functionality.
    
    - Former digest behavior is now encapsulated in instant notifications
      that are sent as soon as the event occurs, with the ability to set a
      rate limit, which digests any events that occur during cool down.
    - Add scheduled notifications that can be received daily, or on specific
      days of the week at a given time.
    - Use a polymorphic data model so that different types of subscriptions
      can be properly managed without type specific code.
    - Move digest sender code into NotificationSender classes.
    - Add tests for new subscription classes, dao, and service.
    
    Refs #8983
Commits on Feb 27, 2019
  1. Upgrade to most recent postgres jdbc version.

    stouffers committed Feb 27, 2019
    The version we were on had not been updated since 2016 and I had
    encountered a bug that was fixed in one of the subsequent versions.
Commits on Feb 21, 2019
  1. Fix array index issue in agenda report service

    stouffers committed Feb 21, 2019
    A recent modification to the BaseAgendaCheckReportService added logic
    that filtered out certain references from the incoming reference list.
    If all references were filtered out, this created an index exception
    later on when the code attempts to access the first reference of the
    list.  This was not a problem before because a ReferenceDataNotFoundEx
    (which is checked in the spotcheck service) was thrown if no references
    were found.
    
    This commit fixes this issue by throwing a ReferenceDataNotFoundEx if
    the filtered reference list is empty.
Commits on Feb 15, 2019
  1. Simplify committee data model in db.

    stouffers committed Feb 15, 2019
    Simplifies the committee data model in the database by removing several
    redundant fields
    
    The main area of simplification is the "reformed" column of the
    committee_version table, which was a redundant value that matched the
    "created" value of the next version's row.  The redundancy technically
    made it more efficient to query for point in time data, but it added
    complexity in the dao layer.
    
    This commit refactors the dao to use just the "created" time for queries
    and computing the "reformed" time using a subquery.  It also simplifies
    some of the update logic and removes a number of unused queries and DAO
    methods.
    
    Fixes #12203
Commits on Feb 13, 2019
  1. Add null check during spotcheck html cleaning

    stouffers committed Feb 13, 2019
    Add a null check in the cleanHtml method in the bill scrape check
    service to prevent it from attempting to parse html from a null string.
  2. Fix spotcheck ordering for int keys.

    stouffers committed Feb 13, 2019
    Fix a bug where integer key values are sorted as if they were strings
    in the spotcheck query.  This is because the int values are stored in an
    hstore and are retrieved as strings.  This is fixed by converting the
    retrieved value to an int when sorting.
Commits on Feb 12, 2019
  1. Standardize spotcheck id columns.

    stouffers committed Feb 12, 2019
    Further standardize the spotcheck id columns that are different for
    different data sources/content types.
    
    - Add year columns for calendars and agendas and split "Bill" column
      into session and print no columns.
    - Use configured columns from the spotcheck table in the detail window
      instead of hardcoded columns.
    
    Fixes #11329
  2. Use queue system for saving api request data.

    stouffers committed Feb 12, 2019
    Implement a queue system for saving api request data to elasticsearch to
    prevent overuse of asynchronous threads.  Previously, each request would
    employ a new thread to save the request data, which was causing
    problems.  Now, request data is saved to an in-memory threadsafe queue,
    which is cleared and persisted in bulk during a scheduled process.
    
    Fixes #12255
  3. Set queue capacity for asynchronous jobs.

    stouffers committed Feb 12, 2019
    Set a queue capacity for the threadpool used to serve asynchronous jobs.
    This will cause exceptions to be raised when the queue becomes too
    large, which will alert us quicker when there is a problem with job
    buildup.
Commits on Feb 8, 2019
  1. Convert 12/19 elastic api log sql script to flyway

    stouffers committed Feb 8, 2019
    Converts the 12/19/18 sql script that drops the request and response
    tables to flyway.  This script was created in the elastic-api-log
    branch, which had not been merged into the dev branch when the existing
    scripts were converted to flyway.  The truncation step is removed and an
    'if exists' clause is added for compatibility with instances that have
    already upgraded to the latest dev code and run the script.
    Also, removing the temporary rules sponsor fix script.
  2. Add 2019 budget bill pdfs

    stouffers committed Feb 6, 2019
    Add pdf files and insert them into the alternate pdfs table for the
    corresponding bills.
    
    Refs #10916
    The budget bill notifier successfully identifed the lack of linked pdfs
    and also stopped notifications when they were added.
Older
You can’t perform that action at this time.