Skip to content

Commit

Permalink
Merge f7432e3 into 55051de
Browse files Browse the repository at this point in the history
  • Loading branch information
konradhalas committed May 12, 2019
2 parents 55051de + f7432e3 commit df2e529
Show file tree
Hide file tree
Showing 9 changed files with 126 additions and 902 deletions.
165 changes: 27 additions & 138 deletions README.md
Expand Up @@ -57,8 +57,7 @@ Dacite supports following features:
- unions
- forward references
- collections
- values casting and transformation
- remapping of fields names
- custom type hooks

## Motivation

Expand Down Expand Up @@ -102,20 +101,13 @@ of `dacite.Config` class

Configuration is a (data) class with following fields:

- `remap`
- `flattened`
- `prefixed`
- `cast`
- `transform`
- `type_hooks`
- `forward references`
- `check_types`

The examples below show all features of `from_dict` function and usage
of all `Config` parameters.

Use a dot-notation path if you want to point a nested data class field in
a configuration, e.g. `"a.b"`. It works for all options.

### Nested structures

You can pass a data with nested dictionaries and it will create a proper
Expand Down Expand Up @@ -236,31 +228,12 @@ result = from_dict(data_class=B, data=data)
assert result == B(a_list=[A(x='test1', y=1), A(x='test2', y=2)])
```

### Forward References

Definition of forward references can be passed as a `{'name': Type}` mapping to
`Config.forward_references`. This dict is passed to `typing.get_type_hints()` as the
`globalns` param when evaluating each field's type.

```python
@dataclass
class X:
y: "Y"

@dataclass
class Y:
s: str

data = from_dict(X, {"y": {"s": "text"}}, Config(forward_references={"Y": Y}))
assert data == X(Y("text"))
```

### Remapping
### Type hooks

If your input data key does not match with a data class field name, you
can use `Config.remap` argument to handle this case. You have to pass
dictionary with a following mapping:
`{'data_class_field': 'input_field'}`
You can use `Config.type_hooks` argument if you want to transform the input
data of a data class field with given type into the new value. You have to
pass a following mapping: `{Type: callable}`, where `callable` is a
`Callable[[Any], Any]`.

```python
@dataclass
Expand All @@ -269,116 +242,36 @@ class A:


data = {
'y': 'test',
'x': 'TEST',
}

result = from_dict(data_class=A, data=data, config=Config(remap={'x': 'y'}))
result = from_dict(data_class=A, data=data, config=Config(type_hooks={str: str.lower}))

assert result == A(x='test')

```
### Flattened

You often receive a flat structure which you want to convert to
something more sophisticated. In this case you can use
`Config.flattened` argument. You have to pass list of flattened fields.

```python
@dataclass
class A:
x: str
y: int


@dataclass
class B:
a: A
z: float


data = {
'x': 'test',
'y': 1,
'z': 2.0,
}

result = from_dict(data_class=B, data=data, config=Config(flattened=['a']))

assert result == B(a=A(x='test', y=1), z=2.0)
```

### Prefixed

Sometimes your data is prefixed rather than nested. To handle this case,
you have to use `Config.prefixed` argument, just pass a following
mapping: `{'data_class_field': 'prefix'}`
If a data class field type is a `Optional[T]` you can pass both -
`Optional[T]` or just `T` - as a key in `type_hooks`. The same with generic
collections, e.g. when a field has type `List[T]` you can use `List[T]` to
transform whole collection or `T` to transform each item.

```python
@dataclass
class A:
x: str
y: int


@dataclass
class B:
a: A
z: float


data = {
'a_x': 'test',
'a_y': 1,
'z': 2.0,
}

result = from_dict(data_class=B, data=data, config=Config(prefixed={'a': 'a_'}))

assert result == B(a=A(x='test', y=1), z=2.0)
```

### Casting
### Forward References

Input values are not casted by default. If you want to use field type
information to transform input value from one type to another, you have
to pass given field name as an element of the `Config.cast` argument
list.
Definition of forward references can be passed as a `{'name': Type}` mapping to
`Config.forward_references`. This dict is passed to `typing.get_type_hints()` as the
`globalns` param when evaluating each field's type.

```python
@dataclass
class A:
x: str


data = {
'x': 1,
}

result = from_dict(data_class=A, data=data, config=Config(cast=['x']))

assert result == A(x='1')
```

### Transformation

You can use `Config.transform` argument if you want to transform the
input data into the new value. You have to pass a following mapping:
`{'data_class_field': callable}`, where `callable` is a
`Callable[[Any], Any]`.
class X:
y: "Y"

```python
@dataclass
class A:
x: str


data = {
'x': 'TEST',
}

result = from_dict(data_class=A, data=data, config=Config(transform={'x': str.lower}))
class Y:
s: str

assert result == A(x='test')
data = from_dict(X, {"y": {"s": "text"}}, Config(forward_references={"Y": Y}))
assert data == X(Y("text"))
```

### Types checking
Expand Down Expand Up @@ -416,14 +309,11 @@ exception. There are a few of them:
with a type of a data class field
- `MissingValueError` - raised when you don't provide a value for a
required field
- `InvalidConfigurationError` - raised when you provide a invalid value
(a field name or a input data key) for a configuration
- `UnionMatchError` - raised when provided data does not match any type
of `Union`
- `ForwardReferenceError` - raised when undefined forward reference encountered in
dataclass


## Development

First of all - if you want to submit your pull request, thank you very much!
Expand Down Expand Up @@ -522,11 +412,10 @@ def products():

```

What if we want to validate our data (e.g. check if `code` has 6 characters) or
use something different than simple built-in types (e.g. we want to use
`Decimal` as a type for `price` field)? Such features are out of scope of
`dacite` but we can easily combine it with one of data validation library.
Let's try with [marshmallow](https://marshmallow.readthedocs.io).
What if we want to validate our data (e.g. check if `code` has 6 characters)?
Such features are out of scope of `dacite` but we can easily combine it with
one of data validation library. Let's try with
[marshmallow](https://marshmallow.readthedocs.io).

First of all we have to define our data validation schemas:

Expand Down
98 changes: 3 additions & 95 deletions dacite/config.py
@@ -1,101 +1,9 @@
from dataclasses import dataclass, field as dc_field, fields, Field
from typing import Dict, Any, Callable, List, Optional, Type

from dacite.data import Data
from dacite.dataclasses import has_field_default_value
from dacite.exceptions import InvalidConfigurationError
from dacite.types import cast_value


class ValueNotFoundError(Exception):
pass
from dataclasses import dataclass, field as dc_field
from typing import Dict, Any, Callable, Optional, Type


@dataclass
class Config:
remap: Dict[str, str] = dc_field(default_factory=dict)
prefixed: Dict[str, str] = dc_field(default_factory=dict)
cast: List[str] = dc_field(default_factory=list)
transform: Dict[str, Callable[[Any], Any]] = dc_field(default_factory=dict)
flattened: List[str] = dc_field(default_factory=list)
type_hooks: Dict[Type, Callable[[Any], Any]] = dc_field(default_factory=dict)
forward_references: Optional[Dict[str, Any]] = None
check_types: bool = True

def validate(self, data_class: Type, data: Data) -> None:
self._validate_field_name(data_class, "remap")
self._validate_data_key(data_class, data, "remap")
self._validate_field_name(data_class, "prefixed")
self._validate_data_key(data_class, data, "prefixed", lambda v, c: any(n.startswith(v) for n in c))
self._validate_field_name(data_class, "cast")
self._validate_field_name(data_class, "transform")
self._validate_field_name(data_class, "flattened")

def make_inner(self, field: Field) -> "Config":
return Config(
remap=self._extract_nested_dict(field, self.remap),
prefixed=self._extract_nested_dict(field, self.prefixed),
cast=self._extract_nested_list(field, self.cast),
transform=self._extract_nested_dict(field, self.transform),
flattened=self._extract_nested_list(field, self.flattened),
check_types=self.check_types,
)

# pylint: disable=unsupported-membership-test,unsubscriptable-object,no-member
def get_value(self, field: Field, data: Data) -> Any:
if field.name in self.flattened or field.name in self.prefixed:
if field.name in self.flattened:
value = data
else:
value = self._extract_nested_dict_for_prefix(self.prefixed[field.name], data)
else:
try:
key_name = self.remap.get(field.name, field.name)
value = data[key_name]
except KeyError:
raise ValueNotFoundError()
if field.name in self.transform:
value = self.transform[field.name](value)
if field.name in self.cast:
value = cast_value(field.type, value)
return value

def _validate_field_name(self, data_class: Type, parameter: str) -> None:
data_class_fields = {field.name for field in fields(data_class)}
for data_class_field in getattr(self, parameter):
if "." not in data_class_field:
if data_class_field not in data_class_fields:
raise InvalidConfigurationError(
parameter=parameter, available_choices=data_class_fields, value=data_class_field
)

def _validate_data_key(self, data_class: Type, data: Data, parameter: str, validator=lambda v, c: v in c) -> None:
input_data_keys = set(data.keys())
data_class_fields = {field.name: field for field in fields(data_class)}
for field_name, input_data_field in getattr(self, parameter).items():
if "." not in field_name:
field = data_class_fields[field_name]
if not validator(input_data_field, input_data_keys) and not has_field_default_value(field):
raise InvalidConfigurationError(
parameter=parameter, available_choices=input_data_keys, value=input_data_field
)

def _extract_nested_dict(self, field: Field, params: Dict[str, Any]) -> Dict[str, Any]:
prefix = field.name + "."
return self._extract_nested_dict_for_prefix(prefix=prefix, data=params)

def _extract_nested_list(self, field: Field, params: List[str]) -> List[str]:
result = []
prefix = field.name + "."
prefix_len = len(prefix)
for name in params:
if name.startswith(prefix):
result.append(name[prefix_len:])
return result

def _extract_nested_dict_for_prefix(self, prefix: str, data: Dict[str, Any]) -> Dict[str, Any]:
result = {}
prefix_len = len(prefix)
for key, val in data.items():
if key.startswith(prefix):
result[key[prefix_len:]] = val
return result
12 changes: 7 additions & 5 deletions dacite/core.py
Expand Up @@ -2,7 +2,7 @@
from dataclasses import fields, is_dataclass
from typing import TypeVar, Type, Optional, get_type_hints, Mapping, Any

from dacite.config import Config, ValueNotFoundError
from dacite.config import Config
from dacite.data import Data
from dacite.dataclasses import get_default_value_for_field, create_instance, DefaultValueNotFoundError
from dacite.exceptions import (
Expand All @@ -20,6 +20,7 @@
is_union,
extract_generic,
is_optional,
transform_value,
)

T = TypeVar("T")
Expand All @@ -36,7 +37,6 @@ def from_dict(data_class: Type[T], data: Data, config: Optional[Config] = None)
init_values: Data = {}
post_init_values: Data = {}
config = config or Config()
config.validate(data_class, data)
try:
data_class_hints = get_type_hints(data_class, globalns=config.forward_references)
except NameError as error:
Expand All @@ -46,15 +46,17 @@ def from_dict(data_class: Type[T], data: Data, config: Optional[Config] = None)
field.type = data_class_hints[field.name]
try:
try:
value = _build_value(
type_=field.type, data=config.get_value(field, data), config=config.make_inner(field)
field_data = data[field.name]
transformed_value = transform_value(
type_hooks=config.type_hooks, target_type=field.type, value=field_data
)
value = _build_value(type_=field.type, data=transformed_value, config=config)
except DaciteFieldError as error:
error.update_path(field.name)
raise
if config.check_types and not is_instance(value, field.type):
raise WrongTypeError(field_path=field.name, field_type=field.type, value=value)
except ValueNotFoundError:
except KeyError:
try:
value = get_default_value_for_field(field)
except DefaultValueNotFoundError:
Expand Down
4 changes: 0 additions & 4 deletions dacite/dataclasses.py
Expand Up @@ -11,10 +11,6 @@ class DefaultValueNotFoundError(Exception):
pass


def has_field_default_value(field: Field) -> bool:
return field.default != MISSING or field.default_factory != MISSING or is_optional(field.type) # type: ignore


def get_default_value_for_field(field: Field) -> Any:
if field.default != MISSING:
return field.default
Expand Down

0 comments on commit df2e529

Please sign in to comment.