New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError: cannot import name 'unquote' #2662

Closed
ilmaruk opened this Issue Jan 15, 2018 · 8 comments

Comments

Projects
None yet
8 participants
@ilmaruk

ilmaruk commented Jan 15, 2018

Long story short

As of yarl 1.0.0 (available since one hour ago), yarl.unquote is no longer available, thus causing an import error at web_urldispatcher.py.

Expected behaviour

Been able to run: from aiohttp import web

Actual behaviour

from aiohttp import web raises the following error:

  File "web.py", line 1, in <module>
    from aiohttp import web
  File "/home/ruggero/Development/mqtt/.venv/lib/python3.6/site-packages/aiohttp/web.py", line 15, in <module>
    from . import (hdrs, web_exceptions, web_fileresponse, web_middlewares,
  File "/home/ruggero/Development/mqtt/.venv/lib/python3.6/site-packages/aiohttp/web_middlewares.py", line 5, in <module>
    from aiohttp.web_urldispatcher import SystemRoute
  File "/home/ruggero/Development/mqtt/.venv/lib/python3.6/site-packages/aiohttp/web_urldispatcher.py", line 21, in <module>
    from yarl import URL, unquote
ImportError: cannot import name 'unquote'

Steps to reproduce

Simply: from aiohttp import web

Your environment

aiohttp 2.3.7 (both)
Ubuntu Linux 16.04
Python 3.6.3

Workaround

Downgrade to yarl 0.18.0

@gcarq

This comment has been minimized.

gcarq commented Jan 15, 2018

Same problem since yarl 1.0.0, downgrading to yarl==0.18.0 helps as workaround.

13:36:06   File "/var/jenkins/workspace/sometest-01-test/test/__init__.py", line 19, in <module>
13:36:06     from aiohttp import web
13:36:06   File "/var/jenkins/shiningpanda/jobs/bd59e0bb/virtualenvs/d41d8cd9/lib/python3.6/site-packages/aiohttp/web.py", line 15, in <module>
13:36:06     from . import (hdrs, web_exceptions, web_fileresponse, web_middlewares,
13:36:06   File "/var/jenkins/shiningpanda/jobs/bd59e0bb/virtualenvs/d41d8cd9/lib/python3.6/site-packages/aiohttp/web_middlewares.py", line 5, in <module>
13:36:06     from aiohttp.web_urldispatcher import SystemRoute
13:36:06   File "/var/jenkins/shiningpanda/jobs/bd59e0bb/virtualenvs/d41d8cd9/lib/python3.6/site-packages/aiohttp/web_urldispatcher.py", line 21, in <module>
13:36:06     from yarl import URL, unquote
13:36:06 ImportError: cannot import name 'unquote'
@panagiks

This comment has been minimized.

Contributor

panagiks commented Jan 15, 2018

Same here, opened a ticket to yarl.

@wojked

This comment has been minimized.

wojked commented Jan 15, 2018

Confirmed @ilmaruk , you were faster 👍

CHANGES
1.0.0 (2018-01-15)
Drop yarl.quote and yarl.unquote public functions (#155)

from: https://pypi.python.org/pypi/yarl

@foxx

This comment has been minimized.

foxx commented Jan 15, 2018

Can we get someone to pin yarl==0.18.0 in deps until the problem is sorted out?

@panagiks

This comment has been minimized.

Contributor

panagiks commented Jan 15, 2018

For reference, #2664

pfreixes added a commit that referenced this issue Jan 15, 2018

@pfreixes

This comment has been minimized.

Contributor

pfreixes commented Jan 15, 2018

@3lnc

This comment has been minimized.

Contributor

3lnc commented Jan 15, 2018

Just in case: I'm working on moving aiohttp internals towards public yarl interfaces.

@fabaff fabaff referenced this issue Jan 15, 2018

Closed

Upgrade yarl to 1.0.0 #11660

4 of 4 tasks complete

asvetlov added a commit that referenced this issue Jan 15, 2018

@3lnc 3lnc referenced this issue Jan 15, 2018

Closed

Compatibility with yarl==1 #2666

0 of 5 tasks complete

openstack-gerrit pushed a commit to openstack-infra/zuul that referenced this issue Jan 15, 2018

James E. Blair
Temporarily pin yarl while aiohttp is broken
Aiohttp had an open requirement specification on yarl which has
released a 1.0 that is backwards incompatible.  Pin to <1.0
until aio-libs/aiohttp#2662 is fixed.

Change-Id: I4e750900501ed92bdbb616f5664f7e8ab7fa99c3

kennyballou added a commit to homecu/cronredux that referenced this issue Jan 15, 2018

Pin dependencies to exact versions
Do not use untested versions at build/deploy time.

For reference: aio-libs/aiohttp#2662.

Signed-off-by: Kenny Ballou <kballou@devnulllabs.io>

@asvetlov asvetlov closed this in dd16789 Jan 15, 2018

@jeremyarr

This comment has been minimized.

jeremyarr commented Jan 16, 2018

Damn that was quick! Good job everyone

@Tobotimus Tobotimus referenced this issue Jan 16, 2018

Closed

Yarl 1.0.0 drops unquote #1258

1 of 2 tasks complete

Jagrmi-C added a commit to Jagrmi-C/osm-validator that referenced this issue Jan 18, 2018

Move to pipenv, fix dependencies
    move from requirements.txt to https://docs.pipenv.org/
    keep locked dependencies on Pipfile.lock
    fix dependencies: aio-libs/aiohttp#2662

openstack-gerrit added a commit to openstack/openstack that referenced this issue Jan 18, 2018

Update git submodules
* Update zuul from branch 'master'
  - Replace master with feature/zuulv3
    
    Change-Id: I5d9e9a573e9bffd6e06c73b2412f12c92169d8a3
    
  - Merge "Stop running tox-cover job" into feature/zuulv3
  - Merge "Link to zuul-base-jobs docs from User's Guide" into feature/zuulv3
  - Merge "Register term_handler for all zuul apps" into feature/zuulv3
  - Merge "Make ZuulDaemonApp an abstract base class" into feature/zuulv3
  - Document connection information for components
    
    Add some information about how the different Zuul components
    communicate with each other.
    
    Change-Id: I637a949e089c8df75203c6e585d6aeb933c2cb98
    
  - Stabilize git driver tests
    
    These tests relied on sleeps which can cause races when running
    the full test suite in parallel.  Instead, wait for the events
    we know will happen to happen.
    
    Also remove the dependency on yarl now that aiohttp has made a
    release which works with yarl 1.0 (however, it does not work with
    <1.0 which is why this needs to be combined with this change to
    fix tests).
    
    Change-Id: Ib1c626cdd3f083dd1d23a3c6547bd7163b66567e
    
  - Merge "Centrally register stack dump handler" into feature/zuulv3
  - Merge "Fix dependency cycle false positive" into feature/zuulv3
  - Merge "Remove unused method term_handler" into feature/zuulv3
  - Fix dependency cycle false positive
    
    This corrects a false-positive in the dependency cycle detection,
    but only for the new URL-style depends-on headers.  It does not
    do so for the legacy gerrit headers.
    
    We used a single history list to store all the changes we enqueued
    ahead of a given change, but this meant that if there was more
    than one path to a change, we would see it in the history on the
    second traversal.  Instead, when traversing the tree, use copies
    of the history list at each stage so that it can be rewound when
    going back up the tree.  The second path to a change will not
    trip the cycle detection, and will proceed on to the point where
    it notices the change is already in the queue and return harmlessly.
    
    Also, check whether the exact change is in the history, not just
    the number, since numbers are no longer unique with multiple sources.
    
    Also, fix a bug in the test_crd_cycle test which was causing the
    test to always pass since the changes were never enqueued due to
    missing approval requirements.
    
    Change-Id: I3241f90a1d7469d433cfa176e719322203d4d089
    Story: 2001427
    Task: 6133
    
  - Merge "Enable direct use of github driver in debug tool" into feature/zuulv3
  - Merge "Add --strip option to encrypt_secret.py" into feature/zuulv3
  - Merge "Handle sigterm in nodaemon mode" into feature/zuulv3
  - Merge "Remove updateChange history from github driver" into feature/zuulv3
  - Merge "Documentation changes for cross-source dependencies" into feature/zuulv3
  - Merge "Add support for protected jobs" into feature/zuulv3
  - Merge "Disambiguate with Netflix and Javascript zuul" into feature/zuulv3
  - Merge "Use hotlink instead log url in github job report" into feature/zuulv3
  - Merge "Really change patchset column to string" into feature/zuulv3
  - Stop running tox-cover job
    
    tox-cover frequently fails and is non-voting so is also frequently
    ignored. Stop running the job until such a time as someone cares enough
    to figure out how to make it stable enough that we might make it voting.
    
    Change-Id: Icfffb8060c80a4cbaa834987654f754bc35fbea0
    
  - Merge "Add cross-source tests" into feature/zuulv3
  - Merge "Support cross-source dependencies" into feature/zuulv3
  - Merge "Add skipped CRD tests" into feature/zuulv3
  - Remove updateChange history from github driver
    
    This is not necessary because updateChange in this driver
    is no longer recursive.
    
    Change-Id: If0322a5938b17000e9ef4106fa4b0ffcbfa80c89
    
  - Documentation changes for cross-source dependencies
    
    Change-Id: Idb6530df5c6e647acd201509d2f0d9d86803db14
    Story: 2001334
    Task: 5885
    
  - Add cross-source tests
    
    Change-Id: Iaf31211d12a2c8ce3b4a2860e079748f7e705aba
    Story: 2001334
    Task: 5885
    
  - Support cross-source dependencies
    
    Additional tests and docs in later patches.
    
    Change-Id: I3b86a1e3dd507fa5e584680fb6c86d35f9ff3e23
    Story: 2001334
    Task: 5885
    
  - Temporarily pin yarl while aiohttp is broken
    
    Aiohttp had an open requirement specification on yarl which has
    released a 1.0 that is backwards incompatible.  Pin to <1.0
    until https://github.com/aio-libs/aiohttp/issues/2662 is fixed.
    
    Change-Id: I4e750900501ed92bdbb616f5664f7e8ab7fa99c3
    
  - Merge "Remove need to start executor as root" into feature/zuulv3
  - Add skipped CRD tests
    
    Change-Id: I145617342b424397ae19cf92335b357e413559ba
    
  - Merge "Move CRD tests to test_gerrit_legacy_crd" into feature/zuulv3
  - Really change patchset column to string
    
    The previous change I26ff56159c2710af1515955d27bf4e9ebfcf76e9 had a
    small bug which rendered it basically a noop. Further it missed some 
    important parts and test changes.
    
    Change-Id: I6cfb8a6af05e589140c2f7c8b1d7228f6d4d8fcb
    
  - Remove need to start executor as root
    
    Now that we have a finger gateway, we no longer need to start the
    executor as root so that the finger streamer on the executor can
    bind to port 79 (default port for the finger streamer is changed
    from 79 to 7900). Remove that requirement.
    
    Change-Id: I6df685044c4ce81fd263043adba832609da100af
    
  - Merge "Add specific setup inventory" into feature/zuulv3
  - Merge "Fix error handling for pidfile" into feature/zuulv3
  - Move CRD tests to test_gerrit_legacy_crd
    
    This new class holds the unit tests which verify the soon-to-be
    "legacy" form of cross-repo-dependencies which use the gerrit
    change-id in the commit message footer.  The new form will use the
    URL instead.  We will maintain both forms for some time, so keep
    these tests around.  In a later change, this file will be duplicated
    to exercise the same behavior with the new URL-based form.
    
    Change-Id: I63a7398640bf82963fadf19acdcd2208c185da0b
    
  - Fix error handling for pidfile
    
    Zuul will now try to open the pidfile and raise an exception if there
    is a permission / issue with the pidfile path.
    
    Change-Id: I3276ffe2de0eeb99a4eb11ba7c0e12a62f1d9886
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>
    
  - Add alembic.ini
    
    When adding a new migration 'alembic revision' required an alembic.ini
    with at least the script location set.
    
    Change-Id: I4704f48229a000b8283802f74d302b53ac5cd080
    
  - Change patchset column to string
    
    When running zuul with Github the sql reporter fails when trying to
    report data [1]. The reason for this is that github uses the git sha of
    the PR head as patchset which is no integer. Thus we must change this
    column to string.
    
    [1] Error message:
      sqlalchemy.exc.DataError: (psycopg2.DataError) invalid input syntax for integer: "fa42019a0d4f2a4a99ed6704c3fd037b7812e08f"
      LINE 1: ...b56049945e792082', 'check', 'foo/bar', 2, 'fa42019a0...
    
    Change-Id: I26ff56159c2710af1515955d27bf4e9ebfcf76e9
    
  - Merge "Reprime the installation map in getGitUrl if needed" into feature/zuulv3
  - Merge "Add the project to GithubUser" into feature/zuulv3
  - Merge "Fix github app authentication in _getNeededByFromPR" into feature/zuulv3
  - Merge "Use configured github server in app mode" into feature/zuulv3
  - Link to zuul-base-jobs docs from User's Guide
    
    In the User's Guide section dealing with Git Repositories, link to
    the zuul-base-jobs documentation to solve half of the TODO comment
    there.
    
    Change-Id: Ic333278f1f00184f033ab01ab4568e53b2e40fea
    
  - Merge "Initialize github auth in getGitUrl if needed" into feature/zuulv3
  - Merge "Correctly use project name in getGitUrl" into feature/zuulv3
  - Use hotlink instead log url in github job report
    
    This change make the job name in the job report comments of github
    driver as a hotlink to instead of using a log url directly. This can
    make the report comments more brief.
    
    Change-Id: I19fb8ffbc153230b7f8eedfcd5ac15ec81a66c72
    
  - Merge "Share a fake pull request database across connections" into feature/zuulv3
  - Merge "Don't treat finger client disconnect as exception" into feature/zuulv3
  - Merge "Fake more of the github3 api" into feature/zuulv3
  - Share a fake pull request database across connections
    
    Because connections can be recreated, ensure that the fake pull
    request database (really a dictionary) is shared across instances
    of connections and fake github classes.
    
    Also, move the fake github3 classes to their own file -- they were
    getting larger and unruly.
    
    Change-Id: I471c1487039c8b25a0bab95d918f31b92b9cd32b
    
  - Fake more of the github3 api
    
    We are currently not testing several GithubConnection methods because
    they are faked out.  Instead, move the fake layer to the github3 api
    so that we exercise our code.
    
    This lets us test methods relating to getting pull requests and
    searching.
    
    Change-Id: Ife355299073e347f8b3270650fdefe55646f6455
    
  - Don't treat finger client disconnect as exception
    
    The fingergw currently logs client disconnects as exceptions.
    This makes the log unnecessarily noisy. Just ignore them.
    
    Change-Id: Ic28acabcb47359d4b7077a1eecddefe0f7094212
    
  - Disambiguate with Netflix and Javascript zuul
    
    There are two other projects named zuul. Let's make sure we disambiguate
    so users aren't confused.
    
    Change-Id: I6c459d062970e2abcbb890d626297595d979d324
    
  - Don't remove builds that don't exist
    
    We saw test_reconfigure_window_fixed test fail in the change to
    switch to Ansible 2.4. This may have been a timing bug where we
    try to remove a build that we've already removed.
    
    Change-Id: Ic55eeb35589b6469b0dcce8267f49eda852a49f7
    
  - Strip \r from build UUID in fingergw
    
    Without stripping this, we won't be able to match UUIDs.
    
    Change-Id: I06b98b7f883433313304bfc3bb21edd5725b94e6
    
  - Handle invalid build UUID in finger gateway
    
    The RPC call will return an empty dict if the build UUID
    cannot be found. We should handle that gracefully.
    
    Change-Id: Ie0fa49e08d9213bf7226c6301896507866c36e28
    
  - Add --strip option to encrypt_secret.py
    
    Sometimes you're storing a password in a file by editting it, but you
    want it to be raw without newlines/whitespace/etc. This lets you do
    that easily.
    
    Change-Id: Idc961b89a5ec3fb639e70a321b4ea587cf743b9d
    
  - Add specific setup inventory
    
    By default, Zuul uses runAnsibleSetup on all inventory nodes prior
    to running job playbooks.
    This translates to doing an 'ansible -m setup' against all nodes, but
    this won't work on nodes where Python is not available, like network
    nodes.
    This change adds a specific setup_inventory.yaml file, which will not contain
    nodes where setup module cannot work.
    
    Change-Id: Ieb02a19036854b8d9089bcd4cc9dd0b46e3ce2fc
    
  - Register term_handler for all zuul apps
    
    Almost all zuul apps use the method term_handler for SIGINT and
    SIGTERM. Defining this centrally in ZuulDaemonApp makes this much
    simpler and without repitition.
    
    Change-Id: I68f8d1bf52b0e16340818d2bcc44cd9fc5868ca7
    
  - Make ZuulDaemonApp an abstract base class
    
    We use ZuulDaemonApp like an abstract base class with run() as an
    abstract method so make that explicit. This creates the groundwork for
    later refactorings like centralized signal handling.
    
    Change-Id: I20f14274df27ab181711b2ca2b80251fa5b09938
    
  - Centrally register stack dump handler
    
    We want the stack dump handler to be present in all zuul apps so this
    can be registered in a central place.
    
    Change-Id: I0c4a97d6ee983aa4d57928682dfb6eeffd050197
    
  - Remove unused method term_handler
    
    This method seems to be superseeded by exit_handler and grep tells me
    it is unused.
    
    Change-Id: I5a4dc126acbbe1ac2f99153bc7757c4f6e46fc8c
    
  - Merge "Add zuul-web tests equivalent to the webapp tests" into feature/zuulv3
  - Merge "Fix misleading message about statsd not installed" into feature/zuulv3
  - Merge "Fix indentation on debug statement" into feature/zuulv3
  - Merge "Add implicit project name matching" into feature/zuulv3
  - Enable direct use of github driver in debug tool
    
    The github debugging tool proved to be quite useful for prototyping
    new stuff using the github api. In the process I ended up copying half
    of the GithubConnection into it. However for trying out new stuff it
    would be great to implement and try out the features/fixes directly in
    the Github driver.
    
    And here it is! The script now starts up a standalone version of the
    Github driver which can be used to query and debug isolated stuff
    against Github.
    
    Change-Id: Ifdad1e69dae009011847869d51ff24000c44adb8
    
  - Fix indentation on debug statement
    
    Change-Id: If8ee9a59e765590eec9a530d85213cc14d8df574
    
  - Really fix canonical/non-canonical project merge
    
    In change Icaf8fca3aa4577b009d691f9a67adcb43ea040f5 the merge of
    canonical and non-canonical projects was fixed. However the fix was
    not complete and only covered the static configuration workflow. The
    dynamic configuration workflow is still broken.
    
    Now really fix by canonicalize the project names in every case and
    make the tenant parameter mandatory.
    
    Change-Id: I5af74763fc9c4be395a341f28c6751d22bd46195
    
  - Add implicit project name matching
    
    Most project pipelines define jobs for the project they are defined
    in. However they still need to name the project explicitly which makes
    actions like repo renaming/movement very difficult.
    
    Thus we want to add the project itself as an implicit fallback.
    
    Change-Id: I273a2b3b1ba2a50565624553e3898be2da5611de
    Depends-On: I5af74763fc9c4be395a341f28c6751d22bd46195
    
  - Add zuul-web tests equivalent to the webapp tests
    
    Change-Id: I5c6337d9f6525df7cbbb2b57c1f709d3e17e796a
    
  - Remove github delay
    
    The caching issues are solved now, so try with no delay again.
    
    Change-Id: I109b88bd3bb2b9ff14f8909e6d03de247c797ca8
    
  - Reprime the installation map in getGitUrl if needed
    
    The getGitUrl is also used by the mergers and executors. In order to
    work with github apps they need to know the mapping of projects to
    installation ids. However as they don't receive webhook events they
    currently only create this mapping on startup. Thus we need to refresh
    the installation map if the project is not found.
    
    Change-Id: I4e0f4473e0402ea0d619b83e56eb2a3f4fbe6698
    
  - Merge "Add github debugging template" into feature/zuulv3
  - Merge "Fix github caching" into feature/zuulv3
  - Add github debugging template
    
    This is a small hacky github debugging template script for testing
    github issues.
    
    Change-Id: I9b8d1e9c2ad85eec0ea81527e97c162e8695124a
    
  - Merge "Use connection type supplied from nodepool" into feature/zuulv3
  - Fix github caching
    
    We do caching of the requests to github using cachecontrol which is
    injected into the urllib3 session. The caching needs to be entirely
    etag based as max-age based caching leads to working with stale data.
    
    Unlike documented [1] cachecontrol doesn't priorize the etag caching
    but doesn't even rerequest until max-age was elapsed.
    
    Thus we need to add a custom caching heuristic [2] which simply drops
    the cache-control header containing max-age. This way we force
    cachecontrol to only rely on the etag headers.
    
    [1] http://cachecontrol.readthedocs.io/en/latest/etags.html
    [2] http://cachecontrol.readthedocs.io/en/latest/custom_heuristics.html
    
    Change-Id: If47e1eaa942914a243fc03666b83cd896665bd71
    
  - Merge "Use python3 for docs publication" into feature/zuulv3
  - Use connection type supplied from nodepool
    
    For supporting windows nodes we need the connection type to be
    configurable. This adds the ansible_connection host variable if
    nodepool defines it.
    
    Change-Id: I6d2f81c7586ae0d533add95ea96a9ea8ce8c3ab5
    
  - Merge "Git driver" into feature/zuulv3
  - Fix docs building
    
    We need graphviz for docs building, use "doc" tag in bindep for it.
    This is needed with updated jobs that look for doc tags.
    
    Change-Id: Idc329ab113761061678c9bd31485ac5e18273e23
    
  - Fix misleading message about statsd not installed
    
    If the stats object is none we actually didn't configure it. In case
    statsd is configured but not installed the scheduler will crash
    earlier with a stack trace.
    
    Change-Id: Id9976d78ca41ed4d2ffb164942048273cff7e07f
    
  - Handle sigterm in nodaemon mode
    
    When running zuul within a container it normally runs in nodaemon mode
    as pid 1. Currently in this mode zuul just ignores SIGTERM which is
    used normally to stop containers. Thus when running within OpenShift
    it waits for a timeout until it gets killed forcefully.
    
    Fix this by handling SIGINT and SIGTERM equally.
    
    Change-Id: I24bd8c953e734fdb9545714126d77cbcdc161bbd
    
  - Use python3 for docs publication
    
    Change-Id: I40ba4939c09d9f3f5105e37a23090dad6cabdbdb
    Depends-On: I2d0222225f4ae6c45d09f9352c256062e2535d63
    
  - Merge "Remove unused function validate_conf" into feature/zuulv3
  - Merge "encrypt_secret: remove the trailing '/' when building url" into feature/zuulv3
  - Add support for protected jobs
    
    For some use cases protected jobs can be useful. Protected jobs can
    only be inherited by jobs defined in the same project. This adds
    support for these protected jobs.
    
    Change-Id: I62a8ecbbfa9eec54ab599bb34148976dcabfd40a
    
  - Merge "Fix doc typo" into feature/zuulv3
  - Merge "Add debug project-pipeline option" into feature/zuulv3
  - Merge "Add implied-branches pragma directive" into feature/zuulv3
  - Merge "Update sphinx jobs to use python3" into feature/zuulv3
  - Fix doc typo
    
    Change-Id: If5eeae9eb87c849272d6dd316e2c174cc9169ef2
    
  - Update sphinx jobs to use python3
    
    The zuul sphinx jobs require python3 to run. Set the sphinx_python
    variable so that it will.
    
    Change-Id: Iee6d1f1003ba9464daffbc0009b82a21001e7739
    Depends-On: I481e032834fdbf674157b2c9a8fa6f95fc570ddb
    
  - Add the project to GithubUser
    
    The data of the GithubUser object is queried lazy in order to save
    requests. In the case of Github apps on GHE we need to supply the
    related project in order to do the lazy requests authenticated as the
    correct installation.
    
    Change-Id: Ic35d870be17eaf99bfe398fadfc122375be0f963
    
  - Fix github app authentication in _getNeededByFromPR
    
    When running as a github app we must supply the project in order to do
    proper authentication as the installation. Otherwise we end up as an
    unauthenticated request which will be rejected by most GHE
    installations.
    
    A draw back currently is that the search will be scoped by the app
    installation (which is either a single repository or an
    organization). As a future improvement we might want to iterate over
    all installation and do the query. However this would need to be
    configurable as with many installations this quickly can become quite
    expensive. A different optimization could be searching in the local
    change cache.
    
    Change-Id: Iee7f009693dc8afa4631d2e48a630bb6da689aec
    
  - Use configured github server in app mode
    
    Currently when running as github app the urls for the api access are
    hard coded to github.com. We need to use the configured github server
    for this if we want to use apps for github enterprise.
    
    Change-Id: I9b98d45f37d7cb6842c95cdce6435b86d33e924d
    
  - Initialize github auth in getGitUrl if needed
    
    The executors and mergers don't initialize the github authentication
    upfront. However they call getGitUrl which gets unauthenticated if we
    run as a github app. So check and initialize the authentication and
    installation map.
    
    Change-Id: If05b9f8660cab1d78ac82ea0455aa24721f3a7e2
    
  - Correctly use project name in getGitUrl
    
    The method getGitUrl is called via the project object and not
    string. While this doesn't matter when adding it to a string this
    breaks lookup of the installation key.
    
    Change-Id: I6632497af0b1e13f6e77b5b7b0a0ee105215a1f9
    
  - Git driver
    
    This patch improves the existing git driver by adding
    a refs watcher thread. This refs watcher looks at
    refs added, deleted, updated and trigger a ref-updated
    event.
    
    When a refs is updated and that the related commits
    from oldrev to newrev include a change on .zuul.yaml/zuul.yaml
    or zuul.d/*.yaml then tenants including that ref is reconfigured.
    
    Furthermore the patch includes a triggering model. Events are
    sent to the scheduler so jobs can be attached to a pipeline for
    running jobs.
    
    Change-Id: I529660cb20d011f36814abe64f837945dd3f1f33
    
  - encrypt_secret: remove the trailing '/' when building url
    
    Change-Id: Ie534063b85d333abfcc4116b4e3903299941f139
    
  - Remove unused function validate_conf
    
    Change-Id: I2a1abcbefb639e82949291f997f6946f136b3dd2
    
  - Merge "Docs: group matchers together and explain them" into feature/zuulv3
  - web: remove 'id' column from builds page
    
    The builds controller doesn't return the 'id' field anymore, this change removes
    the empty column from the web page
    
    Change-Id: I34210c224a42c505795616f4fc6a90eaf80ba69c
    
  - Merge "web: fix key request route" into feature/zuulv3
  - Merge "doc: refine zuul_return instruction" into feature/zuulv3
  - web: fix key request route
    
    By default, aiohttp route identifier matches [^{}/]+, which doesn't
    work for project named with a slash. This change fixes the route so
    that it matches anything until the '.pub' extension.
    
    Change-Id: Ia7d7c257fafd3959e8f13b4d26b7999a156fb5ee
    
  - Add debug project-pipeline option
    
    This may be set by a project to help debug why a job is or is not
    running.  It works speculatively, and so can be used to debug
    a single change.
    
    Change-Id: I1957d21fe7775f786935e5d7d4bdf65b86eb5e4d
    
  - Add implied-branches pragma directive
    
    When two projects have dissimilar stable branch names, for example,
    stable/pike and stable/jewel, but should generally be used together
    and therefore share job variants, it can be difficult to make that
    happen.  Currently, one must add explicit multi-branch matchers
    to every such job and project-template.
    
    This allows a user to add a pragma directive to indicate all the
    jobs in a file should apply to multiple branches.
    
    Change-Id: I57cf159992d8f501cbaf41aef19562951ef6b7ea
    
  - Merge "Fix attribute syntax in docs" into feature/zuulv3
  - Merge "Drop local fork of GitPython for 2.1.8 release" into feature/zuulv3
  - Merge "Send open CORS header for jobs and builds" into feature/zuulv3
  - Merge "Serve keys from canonical project name" into feature/zuulv3
  - Merge "Fix nit in fingergw doc" into feature/zuulv3
  - Merge "Add finger gateway" into feature/zuulv3
  - Fix nit in fingergw doc
    
    Change-Id: I8cff1bef9fd1b84472a8f1d413d7e4cafa9a94b2
    
  - Send open CORS header for jobs and builds
    
    We don't have any precious data yet, so keep builds and jobs open like
    status. This lets us use built artifacts as preview from CI jobs.
    
    Change-Id: I81cad00d2d8f0b763601fc67a4863faa96037fc7
    
  - Docs: group matchers together and explain them
    
    Change-Id: Ib14e8c81edc35e5900b89dd46ba5b3dc4a41fd08
    
  - Add finger gateway
    
    This adds the zuul-fingergw app that should be run as root (so that
    it can connect to the standard finger port 79), but changes user privs
    immediately after binding that port.
    
    Common streaming functions have been moved to streamer_utils.py to
    be shared among modules.
    
    Support for CommandSocket has been included.
    
    Change-Id: Ia35492fe951e7b9367eeab0b145d96189d72c364
    
  - doc: refine zuul_return instruction
    
    The zuul_return task needs to be executed on the executor node,
    otherwise it fails with KeyError: 'ZUUL_JOBDIR'
    
    Change-Id: Ib02648f40b1602c85b8c70ee0965f0fde2d227fe
    
  - Merge "Fix line wrapping in github docs" into feature/zuulv3
  - Merge "Support table prefix for sql reporter" into feature/zuulv3
  - Merge "Remove unused setup_tables" into feature/zuulv3
  - Drop local fork of GitPython for 2.1.8 release
    
    2.1.8 incorporates the noted fixes.
    
    Story: 2001393
    Task: 5982
    Change-Id: I828506bd7c1a1f7ce088e958361782b6cbc71f5a
    
  - Merge "Error on duplicate YAML keys" into feature/zuulv3
  - Merge "Add more job selection debug lines" into feature/zuulv3
  - Error on duplicate YAML keys
    
    This should help avoid a situation where a dictionary value is
    silently overwritten.  This is in accordance with the YAML spec,
    and it is a great mystery why pyyaml doesn't do this already.
    
    Change-Id: I130f2b7e9c9fb8bd4176fd658f378e249a6a3b3f
    
  - Add more job selection debug lines
    
    Even if a global variant matches, a project pipeline variant must
    still match.  This is important information to know when debugging.
    
    Change-Id: Iaaff183c62457f9d264111002371f5f44ced867b
    
  - Merge "Debug job selection" into feature/zuulv3
  - Debug job selection
    
    Add some (well, a lot, more than likely) debug lines for each
    job variant which is considered when freezing the job graph.
    
    This will help admins determine why a job doesn't run.
    
    Change-Id: Ia3b140db2a5d5117f96c2fec9ae1cc9ddd28b2d0
    
  - Fix attribute syntax in docs
    
    The extra column was causing sub-items to render as:
    "tenant.untrusted-projects.<project>:.include".
    
    Change-Id: Ica71a4638a654d6ac140a83ec3bb4a52ca6d5242
    
  - Fix line wrapping in github docs
    
    Change-Id: I96706bfc87dfccb1b619d0def3b895949dbdddb8
    
  - Remove unused setup_tables
    
    Setup tables was called twice.
    
    Change-Id: Idd937e1844689edfadbfeb2e180a8ab497c29ec9
    
  - Support table prefix for sql reporter
    
    In some environments zuul operators may have to rely on external
    database providers. In this case it can be cumbersome to get extra
    databases for each test environment. Adding an optional prefix to the
    table names makes it possible to gracefully run several zuul
    deployments against the same database and ensure they're still
    isolated against each other.
    
    Change-Id: Ib9948d6d74f4dc2453738f5d441e233e39e7f944
    
  - Merge "Add stackdumphandler to zuul-web" into feature/zuulv3
  - Merge "tox: remove validate-layout" into feature/zuulv3
  - Merge "Make all zuul-web urls relative" into feature/zuulv3
  - Remove large status header and tagline
    
    With the dashboard navbar above, the words "Zuul Status" aren't really
    neeed. Also, "Real-time status monitor of Zuul, the pipeline manager
    between Gerrit and Workers." seems like text that's a little out of
    place now.
    
    Change-Id: I5741f1e7edf721eec3a445c054f036a1d17bb172
    
  - Web: Drop some columns from build list
    
    The newrev, node name, and end time are not as useful as the other
    columns, and can be found by following links to the change or logs.
    In the future, we can redesign the display so that the user can select
    extra details for a change, but for now, improve the ability to
    scan the results quickly by removing these less-used columns.
    
    Change-Id: I17179178b5486101c9134f8684c6e69f35e1ea7c
    
  - Serve keys from canonical project name
    
    Rather than asking users to know the 'source' name in order to
    retrieve the project's public key, use the tenant and canonical
    project name.  The tenant already appears in most web urls, and
    the project name should be known to the user.
    
    Change-Id: Icd1269ffdd8879bd177fd452978a2c88b2f1b205
    
  - Make all zuul-web urls relative
    
    Currently some static files are retrieved with relative and some with
    absolute paths. This requires hacks (e.g. ProxyHTMLURLMap) if zuul
    runs under a sub url and makes url rewriting much harder. Switch them
    all to relative paths.
    
    Change-Id: I3350faf760af7b5bd92dbc344a1110dde3a35863
    
  - Merge "Remove implied playbook extensions" into feature/zuulv3
  - Merge "Rename ssh_port to connection_port" into feature/zuulv3
  - Merge "Add command socket support to zuul-scheduler" into feature/zuulv3
  - Merge "Move send_command() into ZuulApp" into feature/zuulv3
  - Merge "Add command_socket setting to executor section" into feature/zuulv3
  - Merge "Add command socket support to zuul-merger" into feature/zuulv3
  - Merge "Correctly stream the remaining buffer" into feature/zuulv3
  - Add command socket support to zuul-scheduler
    
    Bring online commandsocket support for the scheduler.
    
    Change-Id: Ia1719650623e79d40f239776eb770550bb73169b
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>
    
  - Move send_command() into ZuulApp
    
    We can start to remove some duplicate code and make it easier to add
    commandsocket support to other processes.
    
    Change-Id: I605ca36b6627ac4fbea960205b9a88abdeed67f5
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>
    
  - Add command_socket setting to executor section
    
    Like the merger, add support for setting command_socket path via
    zuul.conf.
    
    Change-Id: I88aa47870d98b0906dfb733f68af663c2dc00993
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>
    
  - Merge "Update playbook paths with extension" into feature/zuulv3
  - Add command socket support to zuul-merger
    
    Like we have in zuul-executor, add command socket support for
    zuul-merger.
    
    Change-Id: I66a2cb2ba3f55bdd03e884f47648278e30d2f6ab
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>
    
  - Merge "Remove file extension when building SimpleLayout" into feature/zuulv3
  - Rename ssh_port to connection_port
    
    With the upcoming windows support we don't have ssh as the only
    connection type. As a preparation for this generalize ssh_port to
    connection_port.
    
    Change-Id: Ie505fedea7b641a860b7ac4f834d96ce5351dd74
    Depends-On: Ic1939054f0604411e0122db8dbd7e9886ceaa974
    
  - Fix zuul.projects type in docs
    
    There was a second location where the old "list" type was used in the
    docs that was missed in the change to dict of dicts. Fix this minor
    documentation detail to say dictionary instead of list.
    
    Change-Id: I4c9f90f201040e3044e818b839fc715e906f049d
    
  - Merge "github: add integration documentation" into feature/zuulv3
  - Merge "Add support for shared ansible_host in inventory" into feature/zuulv3
  - Merge "Convert zuul.projects to a dict" into feature/zuulv3
  - Merge "Changes for Ansible 2.4" into feature/zuulv3
  - Merge "Fix mixed canonical/non-canonical project merge" into feature/zuulv3
  - Add stackdumphandler to zuul-web
    
    The stackdumphandler could also be useful for zuul-web from times to
    times.
    
    Change-Id: Icf788391fcf7df94b136ea2d78f72c8d7e5e5778
    
  - web: add /{source}/{project}.pub route
    
    This change adds key:get function to the scheduler gearman worker so
    that zuul-web can request project's public key.
    
    Change-Id: I7ae16f27ccd9efd68c8cf10df7ff0ff5b7b7aa13
    
  - web: make console-stream tenant scoped
    
    This change replaces /console-stream route by /{tenant}/console-stream.
    
    Change-Id: Ib5ecbaf2e8a115bb1edebe71696c565b72e7c481
    Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
    
  - web: add /{tenant}/builds route
    
    This change adds a SqlHandler to query the sql reporter database from
    zuul-web through the /{tenant}/builds.json controller.
    
    This change also adds a /{tenant}/builds.html basic web interface.
    
    Change-Id: I423a37365316cc96ed07ad0895c7198d9cff8be5
    
  - web: add /{tenant}/jobs route
    
    This change adds the 'job:list' job to the scheduler gearman worker
    to expose the tenant jobs list.
    
    This change also adds the /{tenant}/jobs.json endpoint to the zuul-web as well
    as a /{tenant}/jobs.html web interface and command line client:
      zuul show jobs $tenant
    
    Change-Id: I950cb6a809a360867b2daccded9a8a45ac46359c
    
  - Correct exception for missing job dependencies
    
    When a job dependency is not configured to run, the emitted error
    should be clear that's the problem.
    
    Change-Id: I8f696e6d3e82fc88853a7b00f191cc7d458e6a81
    Co-Authored-By: James E. Blair <corvus@inaugust.com>
    
  - Remove implied playbook extensions
    
    We deprecated this a while ago, and most changes to OpenStack
    projects implementing the change have landed.
    
    Change-Id: Ie2519f2b4144761b8e4d17ec3299cf4238e16f08
    
  - Update playbook paths with extension
    
    These playbook paths were missing ".yaml" extentions which fails when
    implied matching goes away (Ie2519f2b4144761b8e4d17ec3299cf4238e16f08).
    
    Change-Id: Iff04d894fffdcec0a0861d2460635b4c44a15ae4
    
  - Remove file extension when building SimpleLayout
    
    When using simple_layout, the setup function is adding ".yaml" to the
    file names it is creating (meaning playbooks get names like
    "playbooks/run/integration.yaml.yaml").
    
    This is hidden by the implied playbook extension matching, which
    searches for files with an added ".yaml" so finds them
    anyway. However, simple_layout unit tests then break when this
    matching is removed (Ie2519f2b4144761b8e4d17ec3299cf4238e16f08).
    
    Change-Id: Ibfacd73840b995873156f1687860ea6792b30b26
    
  - Fix implied branch matchers and tags
    
    Adding an implied branch matcher to jobs on in-repo project defs
    works great when an item *has* a branch.  But some items, such as
    tags, don't.  With recent changes, it is now impossible for a
    project to add a job in-repo that runs in a tag pipeline.
    
    To correct this, we need to drop some of the optimizations which
    assumed we could match the implied branch against existing branch
    matchers, and instead, when adding a job in-repo, simply add a new
    kind of branch matcher, an ImpliedBranchMatcher, that is evaluated
    in a boolean 'and' with any existing branch matchers.
    
    The ImpliedBranchMatcher only fails if the item has a branch, and
    the branch doesn't match.  If the item doesn't have a branch, it
    always succeeds.
    
    This means that when a project adds a job to a tag pipeline in-repo,
    it will most likely only have the ImpliedBranchMatcher, which will
    simply succeed.
    
    It also means that the multiple project configurations present in
    the project's multiple branches can all add jobs to tag pipelines,
    and so to remove such a job, changes may need to be made to all
    branches of a project.  However, there's not much that can be done
    about that at the moment.
    
    Change-Id: Id51ddfce7ef0a6d5e3273da784e407ac72a669db
    
  - Fix complex branch matchers in project configs
    
    A recent change to always apply an implied branch matcher to in-tree
    project configs on projects with branches had an optimization to
    avoid adding unecessary jobs which was too simple.  It assumed that
    if the raw text of the matcher on the job did not match the name
    of the branch, then there was no need to add the job.  That only
    works for branch matchers that are simple branch names.  One that
    is a regex may or may not match the implied branch.
    
    To correct this, when building the job list, run the implied branch
    name against the branch matcher regex (if there is one) for the job
    to determine whether it will not match.
    
    Change-Id: Ibbe097801a45928bc9942991d868a78f5f441887
    
  - tox: remove validate-layout
    
    This is no longer valid
    
    Change-Id: Ie2acf3f0ba982a746c700880614143e6ea9a8ec5
    
  - Merge "Fix scheduler reconfiguration handler" into feature/zuulv3
  - Merge "Don't set job var override_checkout if null" into feature/zuulv3
  - Fix scheduler reconfiguration handler
    
    This method rename was missed in a previous change.
    
    Change-Id: Idfcbc2a600b0933ac1158c612ac754932bd12950
    
  - Merge "Don't shrink windows on reconfiguration" into feature/zuulv3
  - Don't set job var override_checkout if null
    
    It's been suggested this variable would be more convenient to use
    if it were simply omitted rather than set to null if the job does
    not set an override_checkout value.
    
    Also, remove a stray TODO which is done.
    
    Change-Id: Iededec3007857f4aebabcd027c40e9e0fc101ced
    
  - Don't shrink windows on reconfiguration
    
    The window size supplied by the user on a pipeline is really
    the starting window size.  It can grow without bound, and so
    there is no reason to shrink a window on reconfiguration.
    For that matter, it's not necessary to increase it either,
    unless the user has raised the floor higher than the current
    value.
    
    Therefore, use the existing window value on static (ie, dependent)
    change queues during reconfiguration, or the new floor if it is
    higher.
    
    Dynamic queues will get new values from the pipeline (whose value
    is also conditionally updated in the same manner).
    
    In review, we discovered that some folks may be using the expotential
    window type to create a static window.  To make sure this doesn't
    break that case, add a test for it.  Since that can be used to
    intentionally reduce the window size on reconfiguration, this new
    test mirrors the test for window shrinkage added in the previous
    change which must be altered now that we don't auto-shrink on
    reconfiguration.
    
    Change-Id: Iaba25788ebe51ced919dc896aa20aa90675d7773
    
  - Merge "Fix branch checkout order" into feature/zuulv3
  - Merge "Print a message when we start the Zuul console" into feature/zuulv3
  - Print a message when we start the Zuul console
    
    The link for the job console is made available before Ansible starts
    printing things. If the console is empty, users will be greeted with
    an "END OF STREAM" message and will need to keep refreshing until there
    is actual content.
    
    Writing a minimal message to "start" the console will prevent that.
    
    Change-Id: I4200c91b468b1c2fcccee23dda19962b30335ef5
    
  - Convert zuul.projects to a dict
    
    This follows-on from I4476b9d4915d107e29b91229287865bff0ada305 where
    we are converting zuul.projects to a dict for easier access.
    
    With the dependent changes, there should be no in-tree users of
    zuul.projects so we can do this switch.  We will then convert
    zuul._project users back to zuul.projects for the final removal
    (I283e66293110aa3bc99acb1cbc6eb3e0cc11f750).
    
    This updates documentation, and also gives some samples of how to use
    the variables (the "| list" is a bit of a gotcha trick -- python3's
    values() returns a view, so it is necessary for iteration).
    
    Depends-On: Id9a7c137ca5bed25d81087201091157c8401576a
    Depends-On: I9d88f405f34d1c5f75ebf4f52cedfaaab20c3bda
    Change-Id: I3c011f72933e98ccbf8badf0e9197c8659766c51
    
  - Remove nodesets from builds canceled during reconfiguration
    
    We observed errant behavior in the configuration covered by
    test_reconfigure_window_shrink in production when a reconfiguration
    shrunk the active window to less than the current value when
    jobs had already completed.
    
    Correct the underlying issue by removing the nodeset associated with
    a build from the buildset when the reconfiguration routine cancels it.
    Then, if we later launch the same job for some reason, we will obtain
    a new nodeset.
    
    If the build is running at the time it's canceled, we will still need
    the scheduler to return the nodeset to nodepool.  Since it currently
    relies on the value in the buildset to find the nodeset, attach the
    nodeset to the build directly, so that even if we have removed the
    nodeset from the buildset, the scheduler will still have a pointer
    to the nodeset when the build completes.
    
    Having said all that, we really don't want to waste resources by
    shrinking the window on reconfiguration.  A future change is likely
    to correct that and very likely invalidate the test just added.  The
    only other time a build is likely to be canceled during reconfiguration
    yet used again later is if a job is removed, then added back to a
    project while changes are in the queue.  So that we continue to have
    a test which covers this case, add a second test based on that scenario.
    
    Both of these tests fail without the included fix.
    
    Change-Id: If61b34e0f1464cb69d9d0b9053e05f1af996a67b
    
  - Changes for Ansible 2.4
    
    Squashed changes:
       - Use 'inventory' instead of 'hostfile' in ansible.cfg.
    
         'hostfile' is deprecated.
    
       - Use 'os.environ.copy()' in zuul_return.py since this causes 2.4 to
         throw an exception now deep within module.exit_json().
    
    Change-Id: I0a52c9e169a54d24a7b361010045fb10211418b7
    
  - Merge "Add inventory variables for checkouts" into feature/zuulv3
  - Merge "Normalize daemon process handling" into feature/zuulv3
  - Fix mixed canonical/non-canonical project merge
    
    When defining multiple project stanzas for the same project the
    resulting job graphs get merged. This should also work if they are
    given as a mixture of canonical and non-canonical project names like
    the following stanza.
    
    - project:
        name: review.example.com/org/project1
        check:
          jobs:
            - common-config-job
    
    - project:
        name: org/project1
        check:
          jobs:
            - project1-job
    
    However currently only one of the stanzas are effective depending on
    which was processed last.
    
    This can be fixed by canonicalizing the project names when extending
    the unparsed config.
    
    Change-Id: Icaf8fca3aa4577b009d691f9a67adcb43ea040f5
    
  - Normalize daemon process handling
    
    Adopt some of the structure from nodepool to make daemon process
    handling more consistent.  Handle some argument parsing centrally.
    
    Change the default pid file structure to match nodepool:
      /var/run/zuul/<processname>
    
    Attempt to use the pidfile before daemonizing so that errors are
    immediately reported.
    
    Drop the config validation test since it is almost useless at this
    point.
    
    Change-Id: I4a9d9473ce028e0b0cd32a8c48598c1682e1c329
    
  - Merge "Remove unused function toList from scheduler" into feature/zuulv3
  - Merge "Re-enable test_zuul_trigger_project_change_merged" into feature/zuulv3
  - Merge "web: add Cache-Control to static files" into feature/zuulv3
  - Merge "Use username from node information if available" into feature/zuulv3
  - Fix branch checkout order
    
    Since we're deprecating override-branch in favor of override-checkout,
    favor override-checkout when deciding which to use.
    
    Also, change an if to elif to make sure we only do one of these.
    
    Change-Id: I55a94188d9894df27d99416cd3cefff8f3a64718
    
  - github: add integration documentation
    
    Change-Id: Ie3f8f9b68319d3f0c420b8c0770e8fce37f750e1
    
  - web: add Cache-Control to static files
    
    This change add the Cache-Control header to static files' response.
    
    Change-Id: Ibdf1c35bad378507162d807cf5acdf13fc3fab88
    
  - Combine branch templates and pipeline branch matchers
    
    In a situation where a projec-template is defined on multiple branches
    and then used by a project on multiple branches, the behavior was not
    as one might expect.
    
    Currently, assuming no explicit branch matchers are in play, all the
    jobs listed in the project-template will get implied branch matchers
    attached.  So the resulting template looks like
    
     - job @ master
     - job @ stable/newton
     - job @ stable/ocata
    
    Then when that template is applied to the project, since the jobs within
    already have branch matchers, the implied branch matcher for the project
    pipeline definition is not applied.  When the template is added to the
    project on the ocata branch, all 3 of those jobs are added to the project,
    and when it is added on the newton branch, all 3 jobs are added to the
    project again.  That's invisible to the user, until they attempt to remove
    the template from one of the branches.  Because the other branches still
    add the template, which contains all the jobs, they still run.
    
    To correct this, either replace the branch matcher obtained from the job
    in the project template (if it is a simple match of a single branch), or
    combine it using a boolean "and" (if it is something more complex) with a
    branch matcher for the project definition.  When the above template is added
    to a project on the newton branch, only the following job will be added to
    the project-pipeline:
    
     - job @ stable/newton
    
    When the template is added on the ocata branch, likewise only the ocata job
    will be added.
    
    If, instead, the project template had an explicit branch matcher, the
    resulting template might be:
    
     - job @ ^(?!stable/diablo).*$
    
    After adding that to the project on the newton branch, the resulting project
    pipeline would be:
    
     - job @ { ^(?!stable/diablo).*$ AND stable/newton }
    
    Ensuring that since the template was only added to the project on newton,
    its jobs only run on newton changes.
    
    Change-Id: I1969d588bc47b8ab5a54a885a68f98178b16b9d5
    
  - Correctly stream the remaining buffer
    
    In case ansible crashes while executing a command module we loose the
    last line of the log. This is caused because we don't stream the
    remaining buffer but the last line we split out of the buffer.
    
    I wasn't able to construct a test case which makes the original code
    break thus this change is without a test case.
    
    Change-Id: I10ffe248c756e0a77f39f6e075dd3615c53f1b03
    
  - Use username from node information if available
    
    Nodepool knows the username that you should ssh with at image build time
    and includes this information in the node data. Zuul should use this
    username for the executor target.
    
    Change-Id: I1e677061b9fd495b192d25a5825362c81e40d0c6
    Depends-On: Ife0daa79f319aea04ed32513f99c73c460156941
    
  - Add inventory variables for checkouts
    
    Add override_checkout to the inventory variables, along with the
    final checkout used for individual projects.
    
    This can aid certain jobs like devstack which need to alter their
    behavior in complex ways based on the branch the user intends to
    test.
    
    Change-Id: I58d94330e18670052fcd2d92b67a1a091d9249e6
    
  - web: add /static installation instructions
    
    Until a javascript client framework thing such as webpack is there, this change adds
    the instructions to setup the /static directory
    
    Change-Id: Ia2f899af97c3a8d430d2b1712ef26fb040fbbf6c
    
  - Add support for shared ansible_host in inventory
    
    Today it is possible to create the following ansible inventory file:
    
      [foo]
      foo01 ansible_host=192.168.1.1
    
      [bar]
      bar01 ansible_host=192.168.1.1
    
    Which allows a user to create multiple host aliases for a single
    connection. This could be done with ansible groups, however there is
    some functional differences on how ansible runs in that configuration.
    
    We could also request 2 nodes from nodepool, however in this case, it
    would be a waste of CI resources because every alias would need a new
    node from nodepool.
    
    Now, a user is able to alias multiple host names to a single node from
    nodepool by doing the following:
    
      nodeset:
        nodes:
          - name:
              - foo
              - bar
            label: ubuntu-xenial
    
    This would result in a single node request from nodepool, but create
    an inventory file with 2 alaises sharing and single ansible_host
    variable.
    
    Change-Id: I674d6baac26852ee1503feb1ed16c279bf773688
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>
    
  - Merge "Make enqueue-ref <new|old>rev optional" into feature/zuulv3
  - Merge "More documentation for enqueue-ref" into feature/zuulv3
  - Merge "Improve error handling in webapp /keys" into feature/zuulv3
  - Merge "Remove zuul-migrate job" into feature/zuulv3
  - Merge "Switch to threading model of socketserver" into feature/zuulv3
  - Merge "web: add /{tenant}/status route" into feature/zuulv3
  - Merge "web: add /tenants route" into feature/zuulv3
  - Make enqueue-ref <new|old>rev optional
    
    As described in the documentation in
    Ibd7ee306bc461d1597c9c8febcaad89a48b9ff96 the new/old rev arguments
    aren't required in all situations.  For example when triggering a
    periodic job and pretending to be the timer event, you just need the
    refs/head/branch.  For a release/tag replay, I believe you only need
    the newref, so setting them both to zero by default triggers the check
    incorrectly.
    
    Thus we set the arguments to None intially and only if they are
    explicitly set do we check they're not the same.  If they're unset by
    the user, we just set them to zero for the rpc calls.
    
    Change-Id: I8125010efcf8d26dc3a99c87fe2c945314072b72
    
  - More documentation for enqueue-ref
    
    enqueue-ref didn't make it into the client documentation, add it with
    some details about what it does, including some discussion of common
    operations you might perform.
    
    Change-Id: Ibd7ee306bc461d1597c9c8febcaad89a48b9ff96
    
  - Remove tools/trigger-job.py
    
    I don't believe this is submitting jobs correctly at all for v3
    ... while it may be fixable I think effort would be better spent
    getting this into the client if we want the way to trigger a single
    job (the enqueue / enqueue-ref will trigger *all* jobs ... which might
    be enough anyway).  My suggestion is to remove this to avoid
    confusion.
    
    Change-Id: I908657ca57471517a6f706980eaf5f04c15224ed
    
  - web: add /{tenant}/status route
    
    This change adds the 'status:get' job to the scheduler gearman worker
    to expose the formatStatusJSON over gearman.
    
    This change also copies the etc/status/public_html files to the zuul-web
    static.
    
    Change-Id: I46cec321f2e75b92ca02e430135a694216718948
    
  - web: add /tenants route
    
    This change adds the zuul:tenant_list to the scheduler gearman worker
    to expose the list of tenants.
    
    This change also adds the /tenants.json endpoint to the zuul-web as well as
    a web interface to list tenants at / or /tenants.html.
    
    Change-Id: Ia8edb52ec97ebe53205427c828944116eebe03b7
    
  - Merge "Prime github app install map on connection load" into feature/zuulv3
  - Merge "Make encrypt_secret.py work with OpenSSL 0.x" into feature/zuulv3
  - Switch to threading model of socketserver
    
    In an attempt to maybe get more information as to why the finger
    daemon seems to be disappearing on us, try using threads instead
    of forks. Alas, there is no spoon.
    
    Change-Id: I473d874037fb81b940bfe71e5d6d1a17b131635f
    
  - Make encrypt_secret.py work with OpenSSL 0.x
    
    The format was changed in 1.0. This enables Mac OS's default openssl CLI
    tool to work with encrypt_secret.py
    
    Change-Id: Ib5d7a0c5cc6a729bed6fa4a64193444bb48022fb
    
  - Prime github app install map on connection load
    
    When scheduler comes online it will query every repo it knows about for
    the branches of the repo. On GitHub this is done via the API. At this
    point there wasn't a mapping of installation IDs to project names, so
    the regular API was used as opposed to the app specific API, and that
    chewed into the API limits rather quickly.
    
    This change will preemptively query for every install known to the app,
    and then every repo said install has access to, in order to build up the
    map of installation IDs for the projects. It has a side effect of
    priming the cache for app tokens as well. With this change when Zuul
    asks for the branches of the projects it's far more likely that the
    installation_id will be found within the cache.
    
    Change-Id: I2173c089d816fa376ac9ca54addd3529d03e11d1
    Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
    
  - Merge "Fix gerrit branch creation detection" into feature/zuulv3
  - Fix gerrit branch creation detection
    
    We detect when a new branch is created so that we know to perform
    a reconfiguration which will include the new branch.  However,
    the detection for this case in the Gerrit driver still had the
    pre-2.13 form of branch representation.  Update it to support both
    forms, and update the tests to use the new form.
    
    Change-Id: I0480482a59a9f289ca4e449ce70fd1c0872e7999
    
  - Improve error handling in webapp /keys
    
    While investigating some problems with key loading, it was very
    difficult to discern what exactly went wrong while requesting keys. One
    important note is that if a project does not exist, the source drivers
    just add it, which leads to 500 crashes later because those empty
    projects don't have the public_key attribute.
    
    Change-Id: Ic2eb0372620798dd7ca63e9e8205c39bd4629a92
    
  - Merge "On reconfiguration, re-enqueue items at the same position" into feature/zuulv3
  - Merge "Add support for override-checkout, deprecate override-branch" into feature/zuulv3
  - On reconfiguration, re-enqueue items at the same position
    
    Upon reconfiguration, we currently re-enqueue every item back
    into its pipeline, in case a difference in the configuration would
    change what should occur.  In the simple case, this is fine, but
    if a dependent pipeline has a branching dependency structure due
    to already failing jobs, we would erroneously re-order the changes
    back into a linear arrangement.  The next pass through the pipeline
    manager would move those items back to where they should be, but
    in doing so, would reset their builds.
    
    To correct this, after re-enqueueing a change, if the change that
    was previously ahead of it in the pipeline was also successfully
    re-enqueued, immediately move the change behind it (or if the
    change ahead was "None" meaning it was its own head, move it behind
    no change).  This should have the effect of putting changes back
    where they were in relation to other failing changes.  If the change
    ahead was not successfully re-enqueued, the current behavior of
    simply putting it behind the nearest change in the queue is preserved.
    If anything more complex happens, any errors will be corrected on the
    next pass through the pipeline manager.
    
    Change-Id: Ie3771d9bbbc1ca77425cf62751d8e5f70ba1f14c
    
  - Merge "Check start time for wait_time key" into feature/zuulv3
  - Merge "Use user home as work directory of executor" into feature/zuulv3
  - Merge "Increase github delay to 10 seconds" into feature/zuulv3
  - Use user home as work directory of executor
    
    When we config non-root user as executor.user in zuul.conf,
    we should switch to the user's home directory in order to avoid
    "Permission deny" error, sometime we may launch zuul-executor
    process in /root directory.
    
    Change-Id: I20ddeace5822fa58235915d4629f3acd40e2a4b0
    
  - Check start time for wait_time key
    
    If the job didn't get started correctly, build.start_time may be None.
    Check it before calculating the wait_time.
    
    Change-Id: I61cc2d72cb56552ff56e517ef7bc87304d5d6eef
    
  - Add support for override-checkout, deprecate override-branch
    
    We want to support jobs specifying that they check out a tag
    rather than merely a branch.  This accidentally worked in Zuul v2
    with zuul-cloner, and some jobs have come to rely on the behavior.
    There's no reason not to support it, so let's do so.
    
    However, for clarity, change the name of the option to
    "override-checkout".  This is intended to be intuitive (anything
    you can tell git to 'checkout' you can put here), while avoiding
    the suggestion that only branches or tags may be checked out.
    
    The old form, "override-branch" is deprecated and will be removed.
    
    Change-Id: Icc2907e72596626e96d2dc9f6ab1c3026f4085ab
    
  - Remove zuul-migrate job
    
    The job does not work anymore since it tests migration of files in
    project-config that have been removed now.
    
    Depends-On: Ic17bb17222068b1b5cf3ac57c4614af42a679961
    Change-Id: Ide024a72d68112e76c114afe685ea7edf586e013
    
  - Fix syntax with gear unRegisterFunction()
    
    This is to fix the following exception:
    
      2017-10-30 17:02:44,604 ERROR zuul.ExecutorServer: Exception in governor thread:
      Traceback (most recent call last):
        File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 1883, in run_governor
          self.manageLoad()
        File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 1897, in manageLoad
          self.unregister_work()
        File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 1693, in unregister_work
          self.executor_worker.unregisterFunction("executor:execute")
      AttributeError: 'ExecutorExecuteWorker' object has no attribute 'unregisterFunction'
    
    Change-Id: Ib898c4046b83829bf2a42b92897261956bc249bc
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>
    
  - Merge "Use cached branches on tenant reconfiguration" into feature/zuulv3
  - Zuul: add file extension to playbook path
    
    Zuul now supports including the file extension on the playbook path
    and omitting the extension is now deprecrated.  Update references
    to include the extension.
    
    Change-Id: I855444b69e037b1f6a01dac65e700a2a26e8be2b
    
  - Merge "Do late decoding of log stream buffer" into feature/zuulv3
  - Use cached branches on tenant reconfiguration
    
    It takes about one minute for us to query gerrit for all the project
    branches, so don't do that on tenant reconfiguration, except for
    the project which prompted the reconfiguration (in case the event
    was a branch creation or deletion).
    
    Also, clean up some unused arguments to getProject methods.
    
    Change-Id: I625dfc6af00d8d480fcb6bac5d0b86e4846844c0
    
  - Merge "Add multi-branch support for project-templates" into feature/zuulv3
  - Merge "Validate that a job has a run playbook on freeze" into feature/zuulv3
  - Merge "Remove implied run" into feature/zuulv3
  - Merge "Update test fixtures to use explicit run" into feature/zuulv3
  - Add multi-branch support for project-templates
    
    Currently, to avoid confusion, we simply ignore project-template
    definitions after the first.  However, a more intuitive approach
    would be, if the template appears on multiple branches, to have
    it reflect those branches.
    
    To that end, add implied branch matchers to templates in the same
    way that we do to jobs themselves.  This should provide the same
    intuitive behavior where a template defined in zuul-jobs will apply
    to all branches, but one defined in a multi-branch repository will
    add jobs with implied branch matchers.
    
    When a template is defined more than once, combine them (in the same
    way that multiple project definitions are defined) so the resulting
    template contains all the jobs (likely with implied branch matchers).
    
    Allow a template to be defined multiple times (e.g., branches) within
    the same project, but do not allow it to be redefined in another
    project.
    
    Because the multiple project definitions in different branches may
    end up applying the same template (which will have all of the
    per-branch jobs) multiple times, detect duplicate job definitions
    when applying templates and filter them out.
    
    The test test_dynamic_template was originally written to verify that
    a project could not redefine a template defined in another project.
    Clarify that, and update it to support the newly reported error in
    that condition.
    
    Change-Id: I6613885a8c7ebf400e85041e0d68b2eb5ceb033f
    
  - Increase github delay to 10 seconds
    
    In testing with a GitHub Enterprise installation, 3s is sometimes not
    enough for the API to update after an event webhook.
    
    Change-Id: I36a7c465afa6e4232addf6fbc29b8a9c62f41d3f
    
  - Merge "Support file extension in playbook path" into feature/zuulv3
  - Merge "Add pragma directive" into feature/zuulv3
  - Merge "Move test_job_auth_inheritance to test_v3" into feature/zuulv3
  - Merge "Switch to late-binding inheritance" into feature/zuulv3
  - Merge "Add implied branch matchers on 'master'" into feature/zuulv3
  - Merge "Remove test_job_inheritance" into feature/zuulv3
  - Merge "Move test_model.test_job_inheritance_configloader" into feature/zuulv3
  - Do late decoding of log stream buffer
    
    The log stream is read in chunked blocks. When having multi byte
    unicode characters in the log stream it can happen that this character
    is split into different buffers. This can break the decode step with
    an exception [1]. This can be fixed by treating the buffer as binary
    and decoding the final lines.
    
    Further we must expect that the data also contains binary data. In
    order to cope with this further harden the final decoding by adding
    'backslashreplace'. This will replace every occurrence of an
    undecodable character by an appropriate escape sequence. This way we
    can retain all the information (even binary) without being unable to
    decode the stream.
    
    [1]: Log output
    Ansible output: b'Exception in thread Thread-10:'
    Ansible output: b'Traceback (most recent call last):'
    Ansible output: b'  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner'
    Ansible output: b'    self.run()'
    Ansible output: b'  File "/usr/lib/python3.5/threading.py", line 862, in run'
    Ansible output: b'    self._target(*self._args, **self._kwargs)'
    Ansible output: b'  File "/var/lib/zuul/ansible/zuul/ansible/callback/zuul_stream.py", line 140, in _read_log'
    Ansible output: b'    more = s.recv(4096).decode("utf-8")'
    Ansible output: b"UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4094-4095: unexpected end of data"
    Ansible output: b''
    
    Change-Id: I568ede2a2a4a64fd3a98480cebcbc2e86c54a2cf
    
  - Validate that a job has a run playbook on freeze
    
    We now know, before attempting to run a job, whether the inheritance
    hierarchy has produced a main playbook for the job.  If there is none,
    error early.
    
    Also, in the executor, assume that any specified playbooks are
    required to exist, and use the more specific version of the error
    message if they don't.
    
    Change-Id: Id7dc5934c665cf939820b12b5ded53adeb60c0a8
    
  - Remove implied run
    
    With multiple levels of inheritance, the optional implied run
    feature is making it difficult to figure out which jobs run
    playbooks and which don't.  Make the run attribute explicitly
    required to aid in human understanding of the configuration.
    
    Change-Id: Ia8f23bce9898cd4f387554e6787b091b63e75519
    
  - Update test fixtures to use explicit run
    
    Change-Id: I3060a2bf57cef10a5a7ec5299e3491f1f6751221
    
  - Support file extension in playbook path
    
    This allows us to specify the full playbook name, with file extension.
    A documentation update is included in a following patch, as well as
    a deprecation notice for the old form.
    
    Change-Id: I19a3ec50e473f717a7e5e2824f702ad4e6acab02
    
  - Add pragma directive
    
    This allows the user to override the implied branch matcher behavior.
    
    Change-Id: I3ef43fd868988666cb01e8a6bb28552cc42151b4
    
  - Move test_job_auth_inheritance to test_v3
    
    Move this into configuration files so that we can test the
    functionality end-to-end rather than relying on internal APIs
    which are frequently…

silverdaz added a commit to NBISweden/LocalEGA that referenced this issue Jan 22, 2018

Jagrmi-C added a commit to Jagrmi-C/osm-validator that referenced this issue Jan 22, 2018

Move to pipenv, fix dependencies tbicr#29
    move from requirements.txt to https://docs.pipenv.org/
    keep locked dependencies on Pipfile.lock
    fix dependencies: aio-libs/aiohttp#2662
    delete requirements.txt and requirements-dev.txt

DevAlone added a commit to DevAlone/proxy_py that referenced this issue Jan 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment