v3.0.0 #738
Draft
v3.0.0 #738
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* split header encoding/decoding logic out into functions * checkout: simplify methodproxy by not passing worker_ctx in * checkpoint: avoid passing things down through the chain by using a partial * minor tidyup and consistency * checkpoint: standalone rpc proxy reimplemented without "queue consumer" * restore call stack and timeout implementation * test cleanup: proxy retries instead of throwing on connection errors * test cleanup: we now attempt to reconnect forever rather than throwing on the second failure * test cleanup: consumeevent no longer exists * test cleanup: we don't explicitly manage consumers anymore * test cleanup: reply listener disconnection is now very similar to rpc consumer disconnection * longer expiries to ensure queue sticks around until we consume from it * pyflakes * additional test coverage for rpc requests longer than reply queue timeouts * additional test for stopping and restarting standalone proxy * new feature: use uuid of the standalone proxy in its call stack * small refactor to pass queue into reply listener * toxiproxy controller removed "reset" function * add support for raising if reply queue expires with pending replies now that the implementation supports it (standalone rpc proxy case) * import sort * unused import * split header encoding/decoding logic out into functions * additional test coverage for rpc requests longer than reply queue timeouts * introduce "proxy" class replacing ServiceProxy and MethodProxy * squash ServiceProxy and MethodProxy into one object; simplify ReplyListener * unit test coverage for Proxy object * refactor: simplify Proxy object by passing only get_reply rather than whole reply listener * add full publisher "options" to standalone proxy * remove remaining deprecations ahead of 2.9.0 * remove unused helpers * also update docs to remove deprecated param * wait_for_worker_idle is also deprecated * merge fixes * static cleanup * checkpoint: convert messaging.Consumer to use ConsumerMixin directly; small test tweaks to accomoodate * checkpoint: RpcConsumer no longer using QueueConsumer; small test tweaks * move a couple of tests out of test_queue_consumer and into higher-level parts of the codebase * move heartbeat tests back into their own section * test tidyup * checkpoint: reply listener no longer using queue_consumer * include ReplyListener in heartbeat tests * TestDisconnectedWhileWaitingForReply can pass for the service proxy now * remove tests that are now superseded by better disconnection tests * final purge of queue consumer * remove final reference to QueueConsumer in regression test * add regression tests detecting missed heartbeat * don't block consumers on the worker pool * better implementation * checkpoint: DRY things out with a shared ConsumerMixin * tidyup * tidyup; actually include consumer core * tidyup; remove queue consumer tests * import sort * missing import * fix bad super call * fix race condition in standalone rpc proxy check for pending messages * don't need this call anymore * don't need the setters * repr no longer used * nocover entrypoint that isn't executed by test * Revert "fix race condition in standalone rpc proxy check for pending messages" This reverts commit 58e2692. * rpc proxy should redeclare reply queue in case inactive for longer than expiry time; this also fixes the race condition where disconnection-detection determines the reply queue to be expired when in fact it's still being declared (race based on how quickly rabbit responds) * use more reliable passive queue declare rather than the rabbit management api * need absolute import * drop other unreliable uses of rabbit_manager * import sort * cleaner interface for proxy/rpc-call/reply-listener; more inline docs * static fixes * add retry around queue checks to give rabbit some grace time * actually run the retried functions * refactor: use consumer as an object, not a mixin * todo tidyup: remove unnecessary conditional * todo tidyup: no need to pass serializer to responder * todo tidyup: apply patches more consistently * also remove serializer from responder tests * todo tidyup: explict method to stop consumer; it now requeues any subsequently received messages * import sort * more cleanup: remove unnecessary serializer attrs and the incorrect comment referring to kombu * can drop the complex fixture re-ordering now that we don't wait for consumers to close gracefully on kill (this has been fine since we started using a vhost per test) * add tests for new Consumer core * DRY out temporary fixture by moving it to conftest * add tests for container being killed to publisher and event dispatcher * replace existing rpc test for container being killed with one following the same pattern as other entrypoints * fix import * don't need to do this because consumer.run just executes in a managed thread; already tested elsewhere * fix imports * no cover pragmas * tiny bit more cleanup: pass options to publisher earlier * serializer arg not used by consumer * rename "options" to "publisher_options" * import sort * tests for stopping consumers with active workers * accept consumer options in amqp consuming extensions (except rpc, because they all share a consumer) * add autodoc'd modules so we check spelling * spelling fixes * restore AUTO_FIX_IMPORTS * remove HeaderEncoder, HeaderDecoder classes * serialization.setup() returning namedtuple * add comment explaining why we have to use a background thread to wait on the worker pool * be more explicit determining target service/method * speak like yoda i do * upgrade moto (#577)
* target_service now optional on RpcProxy declaration * add preconfigured to wordlist * separate cluster and service clients; rename proxy -> client * use Client postfix in standalone versions; pyflakes * s/XRpcProxy/XRpcClient * s/RpcProxy/ServiceRpc * lot of s/proxy/client
* retry decorator doesn't work on pytest.raises because of rewriting * missing no cover * use a more obvious fix
* fix typo in backwards compat shim * don't eat kwargs that are passed to Entrypoint * test all built-in entrypoints support entrypoint kwargs * support entrypoint kwargs in timer entrypoint * allow 88 character lines
flakey test fix
* Global nameko config
Config made accessible outside container and workers - mainly in the service
definition, e.g.:
```python
from nameko.messaging import Consumer
from nameko import config
class Service:
@consume(
queue=Queue(
exchange=config.MY_EXCHANGE,
routing_key=config.MY_ROUTING_KEY,
name=config.MY_QUEUE_NAME
),
prefetch_count=config.MY_CONSUMER_PREFETCH_COUNT
)
def consume(self, payload):
pass
```
* Fix import ordering
* Add ability to define config options via cli args
Example::
$ nameko run service \
--define AMQP_URI=pyamqp://guest:guest@localhost/yo \
--define YO="{'yo': True}"
* Plain mutable dict is fine as config
* upgrade moto (#577)
* Add config setup and update managers
* No need for user config, a plain dict is fine
* Remove container config
Keep the config pproperty for backward compatibility but raise a warning
if accessed. Also remove config passing from the service runner.
* Migrate testing tooling to use nameko.config
Config fixtures now use config_setup and config_update context managers.
* Migrate container and service runner tests
Now once the test tooling is migrated we can make container and service
runner tests pass
* Upgrade messaging to use nameko.config
* Upgrade utils tests to use nameko.config
* Upgrade generic extensions tests to use nameko.config
* Upgrade RPC to use nameko.config
* Upgrade standalone RPC to use nameko.config
* Upgrade events to use nameko.config
* Upgrade standalone events to use nameko.config
* A few more test upgrades to nameko.config
* Upgrade web extensions to use nameko.config
* Docstring should use nameko.config example
* Upgrade cli moddule to use nameko.config
* upgrade all event tests to work with nameko.config
* don't nullify config in mock_container fixture
* Fix AMQP extensions after v3.0.0-rc merge (and conflicts resolution)
* Fix cli package after v3.0.0-rc merge (and conflicts resolution)
* Check toxiproxy tests
* Left test migration to nameko.config
* Deprecate config dependency provider
* A little PEP8 cleanup
* Fix import sorting
* Stop yielding configs by config fixtures
* Remove unused variable
* Upgrade pylint (a bit) to work well with p3.7
* Migrate serializing to use nameko.config
* Migrate SSL to use nameko.config
* Minor pep8 fixes
* Remove deprecated Config dependency provider from docs
* Deprecate --broker cli option
* Migrate docs examples to use nameko.config
* Remove "maybe" fixmes
* Retry on any error when deleting test rabbit vhosts
This would include connection error which is caught and wrapped into
generic exception. We can get connection refused if sqs requests
mock leaks to the vhost deleting green thread.
* Migrate docs testing examples to use nameko.config
* Finalize disconnection tests upgrade
* Test container config deprecated
* Upgrade test to work with latest kombu versions
This particularly changed:
celery/kombu@1160b92
* Remove forgotten broker default
* Always configure serialization in RPC client...
... even if some options come as arguments.
* More readable config tests
* Add a few comments to test cases
* Remove a fixture use - no need for it
* rename to update_config and setup_config
* Set extra headers for outbound rpc as late as possible Allows for in-entrypoint context data updates. * A small comment added * Publisher and EventDispatcher should also use fresh context data * Update Client docstring
* Better config API * Move `update_config` and `setup_config` helper methods from module and place them on config object itself. * Rename both helper context managers to `patch` and use `clear` argument of `patch` to replace the whole config. * Keep `setup` as simple method only (also straight on `config`) * Show config should show only the simple dict. * Use the new config API * P2.7 compaibility * Update examples * Reuse enter and exit inside decorator * Config patch exception info is not used on exit * Better var naming * Simplified config API dosctring
Minor change to add -c alternative to --config option
Backward compatible passing of custom config to container and runner factories.
Global config improvement
V3.0.0 rc with latest master
* bring GitHub actions (#715) to v3 branch * fix merge error: only one eventlet for py37 * test PRs into all branches
* pin down werkzeug <2 * use a conditional fix for 2.x to be compatible * require packaging lib * import sort * websocket argument was added in 1.0.0 * werkzeug 2.0.1 not published for py2
* fix ReplyListener on standalone rpc with custom publisher_options * update to rc10
#718 fixes the standalone rpc reply listener to honour the amqp uri passed into the rpc client. the reconnection tests have revelealed another bug in the implementation now that the reply listener is using the correct uri. check_for_lost_replies will see a "server unexpectedly closed" OSError when reconnecting after a broken connection. this is because it uses an existing connection from the connection pool, some of which will have previously been connected. the above PR was merged without a green CI build (because _reasons_), so the fix is being addded as a separate commit.
* convert to safe yaml loading * can't safely support lookups from yaml * better use of patch.dict * prevent unsupported recursive env var lookups switching to safe loading changes the behaviour of the default regex parser when you give it a config that contains (unsupported) recursive environment variables. rather than update the tests to assert a different broken behaviour, we now raise a configuration error. this will alert the user that their config file will likely not be parsed correctly, or they're missing the optional library. * skip tests when regex is installed also change description from "recursive" to "nested" * no cover for when skipped
* remove automatic monkey-patching (resolves #703) * warns users if the monkeypatch is not applied * suggests using pytest-eventlet as an alternative # Conflicts: # nameko/testing/pytest.py * static * add nameko test command + update warning + move nameko run patch into cli so it is earlier and more explicit * add __main__ entrypoint so CI can run nameko test * 0.25.1 is oldest supported eventlet on py37+ we need eventlet/eventlet#584 * don't need pytest-eventlet plugin anymore
* support disabling event exchange declarations * fix default, and support auto-delete too * update tests for v3 interface * map None and False to same outcome DECLARE_EVENT_EXCHANGES must be set to False to turn off declarations
* add tests from v2 branch * update to use config.patch * port implementation, tweak existing tests * add test_login_method for amqp/consumer mirrors equivalent for amqp/publisher * static * fix for standalone usecases * import sort
mattbennett
force-pushed
the
v3.0.0-rc
branch
from
c0eb96c
to
5f01a18
Compare
Merge remote-tracking branch 'upstream/master' into v3.0.0-rc # Conflicts: # .github/workflows/ci.yaml # CHANGES # README.rst # docs/examples/test/test_examples.py # nameko/__main__.py # nameko/amqp/publish.py # nameko/cli/commands.py # nameko/cli/run.py # nameko/cli/shell.py # nameko/cli/show_config.py # nameko/cli/utils/config.py # nameko/exceptions.py # nameko/messaging.py # nameko/rpc.py # nameko/standalone/events.py # nameko/standalone/rpc.py # nameko/testing/pytest.py # setup.py # test/amqp/test_publish.py # test/cli/test_main.py # test/standalone/test_event_dispatcher.py # test/standalone/test_rpc_proxy.py # test/test_entrypoints.py # test/test_events.py # test/test_messaging.py # test/test_rpc.py # test/testing/test_utils.py # tox.ini
* Python 3.10 support (#747) * test on py3.10 * only pin down dnspython on compatible versions of python we can remove this pin when we drop support for older eventlet versions * stringify python version * evenlet 0.33.0 needed for py3.10 * release kombu pin; require compatible version on 3.10 * bump pinned version of kombu * fix ssl tests now that kombu correctly checks client certs * correct oldest pin for kombu * import sort * adjust pins * need to be explicit about python versions * 2.7 support stops at kombu 4.6.11 # Conflicts: # setup.py # test/standalone/test_event_dispatcher.py # test/standalone/test_rpc_proxy.py # test/test_events.py # test/test_messaging.py # test/test_rpc.py * simplify tests: use ssl for client and server * import sort * collections module has moved * missing import
* fix flakiness in publish retry tests not clear how this ever passed. it's unhelpfully tied to the kombu implementation which probably means it's a bad test, but just fixing it up for now * upgrade isort * rename whitelist -> allowlist * upgrade moto and pyjwt * bump requests to resolve conflict with boto * pyjwt requires you to specify algorithm now * upgrade urllib to resolve dependency tree * expected retries isn't flakey; depends on version of kombu * update import following deprecation, removes warning * pytest arg --strict renamed to --strict-markers * import sort * no cover pragmas for kombu version conditionals * pyjwt and moto require older versions on py2 * run docs on ubuntu 20.04; drop 3.6 from matrix * put 3.6 back and try all on 20.04 * drop 3.6 again * sort 5.12.0 not available on py3.7 * pin down jinja to a releaase that is still compatible with our old version of sphinx. it's not worth trying to upgrade sphinx because the new docs use mkdocs * importlib-metadata 5.0 is not compatible with latest kombu on py3.7 importlib-metadata is not on later versions of python, so we can use a conditional pin. see # celery/celery#7783 * upgrade pinned version of eventlet fixes https://stackoverflow.com/questions/75137717/eventlet-dns-python-attribute-error-module-dns-rdtypes-has-no-attribute-any * whoops; fix syntax in setup.py * oldest eventlet on py3.10 needs updating too * match "pinned" requests to the one in dev reqs also choose a version that works on all currently supported pythons * refactor conditional to avoid another pragma * copy/paste snafu * try running tests on ubuntu latest
* test on py3.11 too * fix up pylintrc * use older coverage on older python * actually run on 3.11 in actions + use latest actions images to avoid node12 deprecation warnings
drop py2 support
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR bringing the v3 branch back into master.