Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Migrated] [async] Check if args/kwargs are JSON Serializable while running locally #704

Closed
jneves opened this issue Feb 20, 2021 · 3 comments

Comments

@jneves
Copy link
Contributor

jneves commented Feb 20, 2021

Originally from: Miserlou/Zappa#1791 by mariamrf

Context

  • When running an async @task locally, the args and kwargs are passed as they are (even if they're not JSON Serializable).
  • It would be nice to have this check when running locally so we can test for and predict failures of that kind.

Expected Behavior

  • When a @task is run locally with invalid args/kwargs (non-serializable), it should break/raise an exception as this is the actual behavior when it's deployed.

Actual Behavior

  • When a @task is run locally with invalid args/kwargs (non-serializable), it just runs as a normal function and therefore, we can't catch errors of this kind in the tests.

Possible Fix

In async.py:

            lambda_function_name = lambda_function_name_arg or os.environ.get('AWS_LAMBDA_FUNCTION_NAME')
            aws_region = aws_region_arg or os.environ.get('AWS_REGION')

            if (service in ASYNC_CLASSES) and (lambda_function_name):
                send_result = ASYNC_CLASSES[service](lambda_function_name=lambda_function_name,
                                                     aws_region=aws_region,
                                                     capture_response=capture_response).send(task_path, args, kwargs)
                return send_result
            else:
                validate_json_serializable({'args': args, 'kwargs': kwargs})
                return func(*args, **kwargs)

and in utilities.py:

def validate_json_serializable(thing):
    try:
        json.dumps(thing)
    except (TypeError, OverflowError):
        # raise more human-readable error here

Steps to Reproduce

  1. Write a function that accepts a non-serializable type (like an instance of a class)
  2. Run this function locally
  3. Function runs!
  4. Deploy the code to Lambda
  5. Function breaks

Your Environment

  • Zappa version used: 0.45.1
  • Operating System and Python version: 3.6
  • The output of pip freeze:
  • Link to your project (optional):
  • Your zappa_settings.py:
@jottenlips
Copy link
Contributor

I can add these changes if you need contributors, this has bitten us a few times where a dev will pass Django objects, see that it is working locally, and push to release.

@jottenlips
Copy link
Contributor

I had a branch up, but I like your changes more.

@monkut monkut added the has-pr label Jul 27, 2022
@monkut monkut self-assigned this Jul 28, 2022
monkut added a commit that referenced this issue Sep 15, 2022
…unning locally (#1154)

* ✨ add feature (#705)
- JSON unserializable content passed to asyncronous task during local develop will raise exception on purpose to allow developers to catch JSON unserializable errors during development.

:white_check_mark: add `test_async_call_arg_not_json_seralizable` testcase
:fire: remove unnecessary coding comment in `test_async.py`

* :art: fix flake8

* :pencil: fix typo
:wrench: add specific `UnserializableJsonError` exception for clarity

* :white_check_mark: expand testcase to include custome exception, `UnserializableJsonError`.

* :art: run isort/black

* :fire: remove unnecessary assert

Co-authored-by: javulticat <31746850+javulticat@users.noreply.github.com>

* :recycle: simplify `validate_json_serializable()` to accept *args, **kwargs as per review comment

* :fire: remove unnecessary import and use `object()` as unserializable_object sample.

Co-authored-by: javulticat <31746850+javulticat@users.noreply.github.com>
@monkut
Copy link
Collaborator

monkut commented Dec 1, 2022

released in 0.56.0, closing.

@monkut monkut closed this as completed Dec 1, 2022
Ian288 pushed a commit to tackle-io/Zappa that referenced this issue Jul 11, 2023
…ile running locally (zappa#1154)

* ✨ add feature (zappa#705)
- JSON unserializable content passed to asyncronous task during local develop will raise exception on purpose to allow developers to catch JSON unserializable errors during development.

:white_check_mark: add `test_async_call_arg_not_json_seralizable` testcase
:fire: remove unnecessary coding comment in `test_async.py`

* :art: fix flake8

* :pencil: fix typo
:wrench: add specific `UnserializableJsonError` exception for clarity

* :white_check_mark: expand testcase to include custome exception, `UnserializableJsonError`.

* :art: run isort/black

* :fire: remove unnecessary assert

Co-authored-by: javulticat <31746850+javulticat@users.noreply.github.com>

* :recycle: simplify `validate_json_serializable()` to accept *args, **kwargs as per review comment

* :fire: remove unnecessary import and use `object()` as unserializable_object sample.

Co-authored-by: javulticat <31746850+javulticat@users.noreply.github.com>
BarNehemia added a commit to Lightricks/Zappa that referenced this issue Aug 10, 2023
…zappa-0.57.0

* commit '0b1eab14ca39c3a3bfb4e915347e07495171dcba': (27 commits)
  updating workflow actions to remove deprecation warnings (zappa#1243)
  📝 CHANGELOG.md update for 0.57.0 (zappa#1246)
  fixing compatibility with Django 4.2 (zappa#1237)
  Update Readme with patreon and donors (zappa#1234)
  adding support for Python 3.10 (zappa#1231)
  Nose to pytest (zappa#1239)
  lint: updating code style with `make black` (zappa#1238)
  Alternative way to check if running in Docker (zappa#1204)
  📝 CHANGELOG.md update for 0.56.0 release (zappa#1187)
  Improve `get_best_match_zone()` to match by most matched components instead of by length of domain (zappa#1193)
  Bypass python version runtime check when code run in docker (zappa#1180)
  Be able to pass in a batch window when using batch size (zappa#1118)
  Correction to README. (zappa#1177)
  (zappa#908) Update BINARY_SUPPORT to use Content-Encoding to identify if data is binary (zappa#1155)
  Remove special check for Django<1.7, fix for zappa#1158  (zappa#1159)
  Resolve (zappa#410) Logs are missing query strings (zappa#1165)
  Handle spaces in x-forwared-for/remove six (zappa#1127) (zappa#1163)
  add feature (zappa#704) Check if args/kwargs are JSON Serializable while running locally (zappa#1154)
  docs: Add documentation for s3 event object key_filters (zappa#1169)
  Add feature pyenv virtualenv detecting .python-version file (zappa#1153)
  ...

# Conflicts:
#	zappa/cli.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants