Permalink
Switch branches/tags
Commits on Sep 17, 2015
  1. Make it simpler to find decisions

    jabley committed Sep 17, 2015
    Include more context in the ADRs in the file name.
Commits on Jun 16, 2015
  1. Upgrade requests library

    jabley committed Jun 15, 2015
Commits on Jun 9, 2015
  1. Merge pull request #433 from alphagov/remove-STAGECRAFT_DATA_SET_QUER…

    jabley committed Jun 9, 2015
    …Y_TOKEN
    
    Remove STAGECRAFT_DATA_SET_QUERY_TOKEN
Commits on Jun 3, 2015
  1. Merge pull request #431 from alphagov/transformations

    jabley committed Jun 3, 2015
    s/transformations/transforms/ for consistency
Commits on Jun 2, 2015
  1. Merge pull request #430 from alphagov/explain_how_to_run_a_manual_tra…

    jabley committed Jun 2, 2015
    …nsform
    
    [#93436966] Explain how to run a manual transform
Commits on Apr 23, 2015
  1. Add detail to error message

    jabley committed Apr 23, 2015
    When receiving a large amount of data, it can be helpful for a client
    to know which record caused the problem.
Commits on Feb 6, 2015
  1. Merge pull request #419 from alphagov/instrument-transforms

    jabley committed Feb 6, 2015
    Add coarse stats reporting for transforms
  2. Merge pull request #420 from alphagov/fix-missing-day-dvsa-takeup

    jabley committed Feb 6, 2015
    Query for latest data inclusively.
Commits on Jan 15, 2015
  1. Merge branch 'release'

    jabley committed Jan 15, 2015
    Conflicts:
    	backdrop/transformers/dispatch.py
    	tests/transformers/test_dispatch.py
    
    This is an attempt to fix the merge conflict between release and master
    which is causing CI to be be broken.
    
    Yay, force-pushing to master!
Commits on Jan 14, 2015
  1. Initial draft of ADRs

    jabley committed Jan 14, 2015
    See http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions
    
    I’ve tried to avoid hindsight bias as much as possible, and describe why
    we have the current estate. Inevitably though, there will be some.
  2. Merge pull request #405 from alphagov/feature/add-prefix-filter

    jabley committed Jan 14, 2015
    Add filter_by_prefix parameter for regex support
Commits on Jan 12, 2015
  1. Merge pull request #401 from alphagov/add-docs-on-emptying-dataset

    jabley committed Jan 12, 2015
    Add docs on emptying a dataset
Commits on Dec 16, 2014
  1. Merge pull request #389 from alphagov/bugfix/no-data-transforms

    jabley committed Dec 16, 2014
    Don't dispatch task if there is no data
Commits on Nov 11, 2014
  1. Merge pull request #371 from alphagov/fill-data-perf-improvments

    jabley committed Nov 11, 2014
    Fill data perf improvments
Commits on Oct 6, 2014
  1. DRY Request-Id processing

    jabley committed Oct 6, 2014
Commits on Oct 3, 2014
  1. Just use autopep8 as part of the build

    jabley committed Oct 2, 2014
    If a computer is going to moan at me about whitespace, a computer can
    fix it.
Commits on Oct 2, 2014
  1. Add more headers to Access-Control-Allow-Headers

    jabley committed Oct 2, 2014
    We recently added functionality to proxy request headers to backend
    services.
    
    See alphagov/spotlight#813
    
    This code also runs in the browser, so we need to ensure that it isn’t
    restricted by browser security. The other option would be not to try to
    set those headers in the client.
Commits on Sep 23, 2014
  1. Don’t build housekeeping branches in travis

    jabley committed Sep 23, 2014
    Minor change to try to be a good travis citizen and avoid unnecessary
    builds.
    
    The deployed-to-{env} branches are used by us internally to compare what
    we’re about to deploy. Those versions of the code will have already been
    built in travis when we merged the pull request from a topic branch.
  2. Defend against zip bombs

    jabley committed Sep 22, 2014
    A zip bomb is a specially crafted file which expands to take up large
    amounts of CPU time, disk space or memory.
    
    The most well-known example of this is the 42.zip which is a zip file
    consisting of 42 kilobytes of compressed data, containing five layers of
    nested zip files in sets of 16, each bottom layer archive containing a
    4.3 gigabyte (4 294 967 295 bytes; ~ 3.99 GiB) file for a total of 4.5
    petabytes (4 503 599 626 321 920 bytes; ~ 3.99 PiB) of uncompressed
    data.
    
    Since we allow upload of gzip-compressed requests, we should defend
    against malicious attacks like that.
    
    This change adds a SafeGzipDecompressor which can be configured with a
    max buffer size to use, and thus limits the amount of memory that we
    permit a single upload to use.
    
    gzip bomb created by 
    
    ```
    $ dd if=/dev/zero bs=1024 count=102400 | gzip > gzip-bomb.gz
    ```
    
    That creates an empty file full of zeros, which is 100M uncompressed,
    but compressed down to a 100K gzip file.
  3. Add striped rate-limiting

    jabley committed Sep 21, 2014
    Slowness for a single data-set should not result in resource-exhaustion
    for the other data sets. This is a resilience measure.
    
    Allow up to 100 requests per minute, with bursting to 5 requests per
    second.
    
    We are doing this in the application since we have the knowledge about
    URLs in here, rather than in Varnish or nginx.
    
    We need to set rate-limiting high for tests, otherwise the tests fail as
    the rate-limiting kicks in.
  4. Update splinter, add more helpful error diagnosis

    jabley committed Sep 23, 2014
Commits on Sep 9, 2014
  1. Log Request-Id for all incoming requests

    jabley committed Sep 9, 2014
  2. Proxy Request-Id to stagecraft

    jabley committed Sep 9, 2014
    As someone operating a distributed system
    I would like to be able to trace a single request through everywhere
    So that I have good transparency in how the system behaves
Commits on Sep 8, 2014
  1. Simplify conditional

    jabley committed Sep 7, 2014
    Don’t do:
    
    ```
    if true {
      return true
    } else {
      return false
    }
    ```
Commits on Aug 31, 2014
  1. Upgrades requests to 2.4

    jabley committed Aug 31, 2014
    Seems to be a minor issue after they upgraded their vendored version
    of urllib3.
    
    requests/requests#2192
  2. Track dependency versions

    jabley committed Aug 31, 2014
  3. Remove unused code validator subclass

    jabley committed Aug 31, 2014
    This is just an unneeded specialisation of
    ParameterMustBeOneOfTheseValidator.
Commits on Aug 26, 2014
  1. HTTP validation is different to staleness

    jabley committed Aug 26, 2014
    Validation and freshness are distinct concerns for HTTP caches, as
    defined in RFC7234. Separate out the decorators for Cache-Control and
    ETag.
Commits on Aug 16, 2014
  1. Remove duplicated branch test

    jabley committed Aug 16, 2014
Commits on Aug 14, 2014
  1. Remove duplication in the staleness check

    jabley committed Aug 13, 2014
Commits on Jul 30, 2014
  1. Merge pull request #334 from alphagov/revert-333-status_json

    jabley committed Jul 30, 2014
    Revert "Set a 5min cache on the `_status/data-sets` endpoint"
Commits on Jul 28, 2014
  1. Merge pull request #329 from alphagov/convert-excel-float-to-int

    jabley committed Jul 28, 2014
    Attempt to convert excel floats to int
  2. Merge pull request #328 from alphagov/new-data-set

    jabley committed Jul 28, 2014
    Rename NewDataSet to DataSet
Commits on Jun 4, 2014
  1. Upgrade xlrd

    jabley committed Jun 4, 2014
  2. Upgrade statsd

    jabley committed Jun 4, 2014