Adds unwind processor. #167

pwalsh · 2021-09-17T11:00:07Z

TODO:

don't do some stuff when from_key == to_key['name']
support other options on the to_key field spec? Current just name and type
docs in PROCESSORS.md

coveralls · 2021-09-17T11:02:38Z

Pull Request Test Coverage Report for Build 561

28 of 29 (96.55%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 85.081%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
dataflows/processors/unwind.py	27	28	96.43%

Totals
Change from base Build 559:	0.2%
Covered Lines:	2207
Relevant Lines:	2594

💛 - Coveralls

akariv

Left a few comments - also don't forget to update PROCESSORS.md

akariv · 2021-09-24T09:26:15Z

dataflows/processors/unwind.py

+
+    def func(package):
+        matcher = ResourceMatcher(resources, package.pkg)
+        yield package.pkg


Before yielding here, you can process the datapackage itself and remove the from_key field if source_delete and from_key != to_key.

akariv · 2021-09-24T09:30:18Z

dataflows/processors/unwind.py

+from dataflows.helpers.resource_matcher import ResourceMatcher
+
+
+def unwind(from_key, to_key, transformer=None, resources=None, source_delete=True):


I think we should also add here the 'to_key' field (with its type and other options)
(or - if from_key == to_key - update the field's type accordingly)

Adds unwind processor.

9152ddb

akariv requested changes Sep 24, 2021

View reviewed changes

step 1 of managing fields on schema.

a09fdc7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds unwind processor. #167

Adds unwind processor. #167

pwalsh commented Sep 17, 2021 •

edited

coveralls commented Sep 17, 2021 •

edited

akariv left a comment

akariv Sep 24, 2021

akariv Sep 24, 2021

		from dataflows.helpers.resource_matcher import ResourceMatcher


		def unwind(from_key, to_key, transformer=None, resources=None, source_delete=True):

Adds unwind processor. #167

Are you sure you want to change the base?

Adds unwind processor. #167

Conversation

pwalsh commented Sep 17, 2021 • edited

coveralls commented Sep 17, 2021 • edited

Pull Request Test Coverage Report for Build 561

💛 - Coveralls

akariv left a comment

Choose a reason for hiding this comment

akariv Sep 24, 2021

Choose a reason for hiding this comment

akariv Sep 24, 2021

Choose a reason for hiding this comment

pwalsh commented Sep 17, 2021 •

edited

coveralls commented Sep 17, 2021 •

edited