Permalink
Commits on Sep 19, 2018
  1. Fix null vote type issue nysenate.gov bill reports

    McSquirrel committed Sep 19, 2018
    Null vote types were causing exceptions in the nysenate.gov bill when
    used in the compare method of BillVoteId.  This commit makes the
    BillVoteId compare method able to handle null values.
  2. Fix Pipeline error handling.

    McSquirrel committed Sep 19, 2018
    The Pipeline class was not killing the pipeline task workers correctly.
    The error handling method would use CompletableFuture#cancel to stop the
    workers but this did not work as I had expected since CompletableFuture
    does not have any control over the thread that is delivering the future.
    I have successfully implemented the worker killing functionality by
    setting a "finished" flag on the PipelineTask runnable objects.  This
    flag is checked periodically during the PipelineTask excecution loop and
    will break the loop if it is observed to be set.
    
    Additionally, I have placed a 1 hour timeout on the pipeline execution
    in the nysenate.gov bill report service.  This will prevent the report
    from indefinitely blocking the data processor.
    
    Fixes #12180
Commits on Sep 18, 2018
Commits on Sep 17, 2018
  1. Reformat license

    McSquirrel committed Sep 17, 2018
  2. Eliminate npm install warnings

    McSquirrel committed Sep 17, 2018
    Deal with some npm warnings that show up during build to prevent those
    who build our project from being alarmed.
    
    - Add license to package.json
    - Disable auditing for npm install.  We don't execute code from any npm
      packages during runtime.  They are only used as build utilities.
Commits on Sep 14, 2018
  1. Optimize reindexing of transcripts and hearings.

    McSquirrel committed Sep 14, 2018
    Remove redundant loops in transcript/hearing reindex methods.  These
    methods no longer retrieved transcript ids based on the year, so the
    loops were essentially reinexing every single transcript/hearing 1 time
    per year in the loop.
  2. Fix missing subscribe in transcript search service

    McSquirrel committed Sep 14, 2018
    The handleRebuildEvent method in ElasticTranscriptSearchService was
    missing the @subscribe annotation necessary to subscribe the method to
    events passed through the event bus.  This commit adds the annotation.
  3. Adjust elasticsearch index shard numbers.

    McSquirrel committed Sep 14, 2018
    Adjusts the number of shards assigned to each index.  All indices have 1
    shard by default.  High volume indices (in terms of documents, size,
    and/or usage) override the default and are assigned more shards.
  4. Add concurrent reindexing for bill search.

    McSquirrel committed Sep 14, 2018
    Modifies the bill reindex method to use several threads to load bills
    and index them into elasticsearch.  This is recommended by elasticsearch
    documentation, and makes the reindex a bit faster.
Commits on Sep 13, 2018
  1. Split large elasticsearch bulk requests.

    McSquirrel committed Sep 13, 2018
    Add logic to the method that posts bulk requests to split abnormally
    large bulk requests.  Bulk requests over a specific size get divided
    into smaller bulk requests which are then posted in sequence.
Commits on Sep 11, 2018
  1. Remove replicas from default es index settings.

    McSquirrel committed Sep 11, 2018
    Make all elasticsearch indices have 0 replicas by default, because we
    do not run multiple nodes.
  2. Address memory issues when reindexing bills.

    McSquirrel committed Sep 11, 2018
    Makes some adjustments to the code that reindexes bills to address out
    of memory issues that occur in elasticsearch.
    - Reduced batch size for bill reindexing.
    - Turn off index refresh for bill reindexing.
  3. Modifications to notifications.

    McSquirrel committed Sep 11, 2018
    Some assorted fixes/modifications to notifications for the ES6 upgrade:
    - Change mechanism of notification id generation so that it works.
    - Change indexed notification fields so that they match view fields.
    - Alter method that gets list of notifications so that it uses a
      structured query rather than a query string.
    - Add unit tests to ensure there are no cycles in the NotificationType
      hierarchy.
Commits on Sep 7, 2018
  1. Handle event bus errors by posting notifications

    McSquirrel committed Sep 7, 2018
    Currently, event bus exceptions are logged and not propagated.
    I have created a new exception handler for the event buses that posts a
    notification of the error.
Commits on Sep 5, 2018
  1. Refactor Elasticsearch Daos

    McSquirrel committed Sep 5, 2018
    Some significant restructuring of the elasticsearch daos:
    
    - Combine methods to get search request, response, and result in base
      elasticsearch dao.
    - Instead of throwing an exception when querying an empty index, return
      an empty response.
    - Move all uses of RestHighLevelClient to the base elasticsearch dao.
    - Handle all IOExceptions thrown by the RestHighLevelClient by
      rethrowing as a runtime exception.
    - Reorganized some additional code in search daos to look nicer.
Commits on Sep 4, 2018
  1. Adjust elastic search client config.

    McSquirrel committed Sep 4, 2018
    Some small adjustments to the elastic search client config:
    - Check the boolean result of the client's ping method.
    - Remove null return on connection failure, throw ex instead.
Commits on Aug 31, 2018
  1. Rethrow exceptions in elastic base dao.

    McSquirrel committed Aug 31, 2018
    There were some IOExceptions in elastic base dao that were caught and
    logged without additional handling.  I have modified these catches to
    rethrow the exceptions as ElasticsearchExceptions so they interrupt the
    code that invoked the search methods and can be handled by our
    generalized exception handlers down the stack.
Commits on Aug 30, 2018
  1. Fix home page layout on mobile

    KevinCaseiras committed Aug 30, 2018
    fixes #11972
Commits on Aug 29, 2018
  1. Fix bug in saved spotcheck mismatch retrival.

    McSquirrel committed Aug 29, 2018
    There was a bug present in the query to retrieve spotcheck mismatches
    for specific content keys for the purpose of determining resolved
    mismatches for a report.  The query was attempting to match the string
    value of key hstores with the output of the java dao function for
    generating hstore strings.  There is no guarantee that these string
    values will be equal, even if the java hstore generator creates values
    that can be parsed into hstores.  To fix this, the query has been
    modified so that passed in key parameters are converted to hstore and
    then matched with stored keys as hstores.
Commits on Aug 23, 2018
  1. Imports optimized.

    JacobKeegan committed Aug 23, 2018
  2. Pulled from dev.

    JacobKeegan committed Aug 23, 2018
  3. Several small changes.

    JacobKeegan committed Aug 23, 2018
Commits on Aug 21, 2018
  1. Simplified some error-checking in ApplicationConfig.

    JacobKeegan committed Aug 21, 2018
    All SearchDao's now convert to Json using the ES Json converter I made.