Use marshmallow under the hood #61

sloria · 2015-09-20T21:26:36Z

Proposal

Replace Arg objects with marshmallow fields. Use marshmallow for validation.

Why?

Webargs and marshmallow have redundant functionality for validating input data against a user-defined schema. Marshmallow does a better job of than webargs in a number of respects:

Nested validation is much more powerful in marshmallow. marshmallow's Nested field would solve allow_missing=False not working with nested args #39 and allow_missing=True not working with nested args on passing null #51.
Marshmallow does error bundling. Solves Add "bundle_errors" options to send multiple errors for one parse #58
Marshmallow handles null and missing inputs in a more predictable way.
Marshmallow has more consistent error messaging.
Marshmallow's validation is more stable and well-tested

Proposed API

from webargs import use_kwargs, fields


@use_kwargs({
    'title': fields.Str(required=True, location='json'),
    'description': fields.Str(required=False),
})
def myview(title, description):
    return 'title: {}, description: {}'.format(title, description)


# with nesting
@use_kwargs({
    'data': fields.Nested({
        'title': fields.Str(),
        'description': fields.Str(),
    })
})
def myview(data):
    return 'title: {}, description: {}'.format(**data)


# respects load_from
@use_kwargs({
    'x_custom_header': fields.Str(load_from='X-Custom-Header')
})
def myview(x_custom_header):
    return 'Header: {}'.format(x_custom_header)


# Passing in a Schema
from marshmallow import Schema

class PostSchema(Schema):
    title = fields.Str(required=True, location='json')
    description = fields.Str(required=False)
    id = fields.Int(dump_only=True)  # won't be included in parsed output

@use_kwargs(PostSchema)
def myview(title, description):
    return 'title: {}, description: {}'.format(title, description)

The text was updated successfully, but these errors were encountered:

marcellarius · 2015-09-20T22:19:42Z

This seems like a good solution to me.

On Mon, Sep 21, 2015 at 9:26 AM, Steven Loria notifications@github.com
wrote:

Proposal

Replace Arg objects with marshmallow fields. Use marshmallow for
validation.
Why?

Webargs and marshmallow have redundant functionality for validating input
data against a user-defined schema. Marshmallow does a better job of than
webargs in a number of respects:

Nested validation is much more powerful in marshmallow.
marshmallow's Nested field would solve allow_missing=False not working with nested args #39
allow_missing=False not working with nested args #39 and allow_missing=True not working with nested args on passing null #51
allow_missing=True not working with nested args on passing null #51.

Marshmallow does error bundling. Solves Add "bundle_errors" options to send multiple errors for one parse #58
Add "bundle_errors" options to send multiple errors for one parse #58

Marshmallow handles null and missing inputs in a more predictable
way.

Marshmallow has more consistent error messaging.

Marshmallow's validation is more stable and well-tested

Proposed API

from webargs import use_kwargs, fields

@use_kwargs({
'title': fields.Str(required=True, location='json'),
'description': fields.Str(required=False),
})def myview(title, description):
return 'title: {}, description: {}'.format(title, description)

with nesting@use_kwargs({
'data': fields.Nested({
    'title': fields.Str(),
    'description': fields.Str(),
})
})def myview(data):
return 'title: {}, description: {}'.format(**data)

respects load_from@use_kwargs({
'x_custom_header': fields.Str(load_from='X-Custom-Header')
})def myview(x_custom_header):
return 'Header: {}'.format(x_custom_header)

Passing in a Schemafrom marshmallow import Schema

class PostSchema(Schema):
title = fields.Str(required=True, location='json')
description = fields.Str(required=False)
id = fields.Int(dump_only=True) # won't be included in parsed output
@use_kwargs(PostSchema)def myview(title, description):
return 'title: {}, description: {}'.format(title, description)

—
Reply to this email directly or view it on GitHub
#61.

hello-josh · 2015-09-21T00:12:03Z

+1

sloria · 2015-09-22T02:51:18Z

Making lots of progress on this.

I haven't yet decided where fields should be imported from.

Here are the alternatives (note that Nested is a webargs-specific field):

(1) Import from webargs.fields

from webargs import fields
from webargs.flaskparser import use_args
from marshmallow import validate

@use_args({
    'data': fields.Nested({
        'id': fields.Int(),
        'email': fields.Str(validate=validate.Email()),
    })
})

(2) Import from webargs

import webargs as wa
from webargs.flaskparser import use_args
from marshmallow import validate

@use_args({
    'data': wa.Nested({
        'id': wa.Int(),
        'email': wa.Str(validate=validate.Email()),
    })
})

(3) Import from marshmallow

from webargs import Nested  # or from webargs.fields import Nested
from webargs.flaskparser import use_args
from marshmallow import fields, validate

@use_args({
    'data': Nested({
        'id': fields.Int(),
        'email': fields.Str(validate=validate.Email()),
    })
})

Any preferences?

marcellarius · 2015-09-22T03:22:01Z

or 3). I think option 3) makes it clear that it's using marshmallow.

On Tue, Sep 22, 2015 at 2:51 PM, Steven Loria notifications@github.com
wrote:

Making lots of progress on this.

I haven't yet decided where fields should be imported from.

Here are the alternatives (note that Nested is a webargs-specific field):

(1) Import from webargs.fields

from webargs import fieldsfrom webargs.flaskparser import use_argsfrom marshmallow import validate
@use_args({
'data': fields.Nested({
'id': fields.Int(),
'email': fields.Str(validate=validate.Email()),
})
})

(2) Import from webargs

import webargs as wafrom webargs.flaskparser import use_argsfrom marshmallow import validate
@use_args({
'data': wa.Nested({
'id': wa.Int(),
'email': wa.Str(validate=validate.Email()),
})
})

(3) Import from marshmallow

from webargs import Nestedfrom webargs.flaskparser import use_argsfrom marshmallow import fields, validate
@use_args({
'data': Nested({
'id': fields.Int(),
'email': fields.Str(validate=validate.Email()),
})
})

Any preferences?

—
Reply to this email directly or view it on GitHub
#61 (comment).

sloria · 2015-09-22T03:42:39Z

Fair point. On the other hand, perhaps we don't want to "leak" the marshmallow abstraction? As a user, I don't really like being required to import a 3rd-party library's (marshmallow) modules in the common case, but that's personal preference.

sloria · 2015-09-22T03:45:25Z

From discussion with @jmcarp : "I think i prefer 1 or 2 just so it's harder to use the wrong Nested".

hello-josh · 2015-09-22T11:29:06Z

I would argue that you shouldn't import from Marshmallow at all. If you want to use Marshmallow under the good, go all the way.

from webargs import fields, validate

sloria · 2015-09-22T12:44:43Z

To play devil's advocate: there's a risk of hiding too much. If users import Schema from webargs, they might think that webargs modifies behavior of vanilla marshmallow Schemas in some way.

That said, I think importing fields and validate from webargs is fine.

sloria · 2015-09-22T13:15:01Z

There's also an argument for future-proofing: we can always add imports to webargs without breaking backwards-compat; we cannot remove them without breaking compat.

See discussion on #61 (comment)

sloria · 2015-09-27T15:43:02Z

The marshmallow branch is merged. We'll have a release out later today.

hello-josh · 2015-09-27T19:28:58Z

👍

This adds a custom `Date` field which respects `settings.TIMEZONE`. This also exposes elasticsearch_dsl's fields from the top-level module, so that all fields are imported from the same place: ```python from elasticsearch_metrics import Keyword, Date ``` This approach was the same used in webargs, which uses marshmallow fields under the hood but overrides a single field. Discussion: marshmallow-code/webargs#61 (comment) close #18

This adds a custom `Date` field which respects `settings.TIMEZONE`. This also exposes elasticsearch_dsl's fields from the metrics module, so that all fields are imported from the same place: ```python from elasticsearch_metrics import Keyword, Date ``` This approach was the same used in webargs, which uses marshmallow fields under the hood but overrides a single field. Discussion: marshmallow-code/webargs#61 (comment) close #18

This adds a custom `Date` field which respects `settings.TIMEZONE`. This also exposes elasticsearch_dsl's fields from the metrics module, so that all fields are imported from the same place: ```python from elasticsearch_metrics import metrics class MyMetric(metrics.Metric): my_keyword = metrics.Keyword() my_date = metrics.Date() ``` This approach was the same used in webargs, which uses marshmallow fields under the hood but overrides a single field. Discussion: marshmallow-code/webargs#61 (comment) close #18

sloria mentioned this issue Sep 20, 2015

Marshmallow and webargs working together marshmallow-code/apispec#12

Closed

sloria mentioned this issue Sep 21, 2015

Use marshmallow to define args and do validation #62

Merged

12 tasks

sloria added a commit that referenced this issue Sep 25, 2015

Make fields import-able from webargs.fields

aad83d2

See discussion on #61 (comment)

sloria added a commit that referenced this issue Sep 25, 2015

Make fields import-able from webargs.fields

b488c15

See discussion on #61 (comment)

sloria closed this as completed Sep 27, 2015

frol mentioned this issue Nov 14, 2015

Migration to webargs noirbizarre/flask-restplus#9

Open

fernandojunior mentioned this issue Jul 1, 2016

Replace reqparse for webargs fernandojunior/anerp#1

Open

sloria mentioned this issue Aug 20, 2018

Make fields importable from the metrics module and add Date CenterForOpenScience/django-elasticsearch-metrics#38

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use marshmallow under the hood #61

Use marshmallow under the hood #61

sloria commented Sep 20, 2015

marcellarius commented Sep 20, 2015

with nesting@use_kwargs({

respects load_from@use_kwargs({

Passing in a Schemafrom marshmallow import Schema

hello-josh commented Sep 21, 2015

sloria commented Sep 22, 2015

marcellarius commented Sep 22, 2015

sloria commented Sep 22, 2015

sloria commented Sep 22, 2015

hello-josh commented Sep 22, 2015

sloria commented Sep 22, 2015

sloria commented Sep 22, 2015

sloria commented Sep 27, 2015

hello-josh commented Sep 27, 2015

Use marshmallow under the hood #61

Use marshmallow under the hood #61

Comments

sloria commented Sep 20, 2015

Proposal

Why?

Proposed API

marcellarius commented Sep 20, 2015

with nesting@use_kwargs({

respects load_from@use_kwargs({

Passing in a Schemafrom marshmallow import Schema

hello-josh commented Sep 21, 2015

sloria commented Sep 22, 2015

marcellarius commented Sep 22, 2015

sloria commented Sep 22, 2015

sloria commented Sep 22, 2015

hello-josh commented Sep 22, 2015

sloria commented Sep 22, 2015

sloria commented Sep 22, 2015

sloria commented Sep 27, 2015

hello-josh commented Sep 27, 2015