Py 2to3 #420

cormachallinanderilinx · 2023-12-19T18:17:42Z

No description provided.

EricSoroos · 2024-01-08T11:55:06Z

ckanext/iati/archiver.py

@@ -292,14 +292,14 @@ def archive_package(package_id, context, consecutive_errors=0):
        else:
            new_extras['data_updated'] = None

-        for key, value in new_extras.iteritems():
-            if value and (key not in extras_dict or unicode(value) != unicode(extras_dict.get(key, ''))):
+        for key, value in list(new_extras.items()):


Generally, you don't need the list() when iterating over it. 2to3 adds this on .items() (and keys & values) just to be sure.

EricSoroos · 2024-01-08T11:56:06Z

ckanext/iati/helpers.py

@@ -124,7 +124,7 @@ def get_publisher_obj_extra_fields(group_dict):
    }

    for ex in group_dict.get("extras", []):
-        if ex.get("key", None) in formatter_map.keys():
+        if ex.get("key", None) in list(formatter_map.keys()):


EricSoroos · 2024-01-08T11:56:12Z

ckanext/iati/helpers.py

@@ -139,7 +139,7 @@ def get_publisher_obj_extra_fields_pub_ids(group_dict):
        'publisher_first_publish_date': render_first_published_date_parse
    }
    for ex in group_dict:
-        if ex in formatter_map.keys():
+        if ex in list(formatter_map.keys()):


EricSoroos · 2024-01-08T11:57:11Z

ckanext/iati/logic/action.py

@@ -75,7 +75,7 @@ def package_create(context, data_dict):
    created_package = create_core.package_create(context, data_dict)

    # Part of first publisher date patch - after package create patch the organization
-    if 'owner_org' in data_dict.keys():
+    if 'owner_org' in list(data_dict.keys()):


Suggested change

if 'owner_org' in list(data_dict.keys()):

if 'owner_org' in data_dict:

EricSoroos · 2024-01-08T11:57:23Z

ckanext/iati/logic/action.py

@@ -280,7 +280,7 @@ def issues_report_csv(context, data_dict):
        result = logic.get_action('package_search')(context, data_dict)
        if result['count'] > 0:
            publishers = result['facets']['organization']
-            for publisher_name, count in publishers.iteritems():
+            for publisher_name, count in list(publishers.items()):


EricSoroos · 2024-01-08T12:02:02Z

ckanext/iati/views/admin.py

@@ -50,7 +50,7 @@ def _active_publisher_data(from_dt, to_dt):
    log.info(query)
    conn = model.Session.connection()
    rows = conn.execute(query)
-    active_publishers = [{key: value for (key, value) in row.items()} for row in rows]
+    active_publishers = [{key: value for (key, value) in list(row.items())} for row in rows]


EricSoroos · 2024-01-08T12:02:56Z

ckanext/iati/views/dashboard.py

@@ -1,5 +1,5 @@
 from flask import Blueprint
-from ckan.lib.base import request, response, render, abort
+from ckan.lib.base import request, render, abort


Is this intended?

EricSoroos · 2024-01-08T12:04:13Z

ckanext/iati/views/spreadsheet.py

                iati_keys = dict([(f[2], f[0]) for f in PublisherRecordsUpload.CSV_MAPPING])
-                for key, msgs in e.error_dict.iteritems():
+                for key, msgs in list(e.error_dict.items()):


EricSoroos · 2024-01-08T12:04:52Z

ckanext/iati/views/spreadsheet.py

@@ -1,5 +1,5 @@
 from flask import Blueprint, make_response
-from ckan.lib.base import response, render, abort
+from ckan.lib.base import render, abort


EricSoroos · 2024-01-08T12:04:57Z

ckanext/iati/views/reports.py

@@ -1,5 +1,5 @@
 from flask import Blueprint, make_response
-from ckan.lib.base import request, response, render, abort
+from ckan.lib.base import request, render, abort


EricSoroos · 2024-01-09T16:57:49Z

ckanext/iati/archiver.py

-    with open(saved_file, 'r') as f:
-        content = f.read()
-    content = re.sub(r'generated-datetime="[^"]+"', '', content)
+    with open(saved_file, 'rb') as f:


I think these were actually equivalent, because opening in text mode assumes UTF-8 character encoding.

But then the hash needs to have binary data, so you could either revert this bit and do the encode on 538, or just do the hash of content_bytes, so you didn't have to encode again.

EricSoroos · 2024-01-09T17:00:24Z

ckanext/iati/logic/csv_action.py

@@ -179,7 +180,12 @@ def json(self):
        _org_data = PublishersListDownload._get_publisher_data()
        for org in _org_data:
            if org.Group.state == 'active' and int(org.package_count) > 0:
-                json_data.append(OrderedDict(list(zip(self._headers, self._prepare(org)))))
+                ordered_dict = OrderedDict(zip(self._headers, self._prepare(org)))
+                for key, value in ordered_dict.items():


This would make sense to do in the _prepare method, I think.

Actually, I'm really unclear how that can end up as bytes, since it should be an array?

EricSoroos · 2024-01-09T17:28:07Z

ckanext/iati/views/spreadsheet.py

@@ -226,7 +226,8 @@ def csv_upload_datasets():
        vars['file_name'] = csv_file.filename
        data = io.BytesIO(_data)


This looks like a complicated version of:

reader = csv.DictReader(data) for row in reader: task={} task['title'] = row.get('title', '') or "No Title" task['task_id'] = str(uuid.uuid4()) job = jobs.enqueue(records_upload_process, [json.dumps([row], ensure_ascii=False).encode('utf-8'), c.user]) task['task_id'] = str(job.id) tasks.append(json.dumps(task))

Except I really don't understand enqueuing an array of [bytes, object], where the bytes is a json string of an array of a single dict. But I guess that's what's required upstream.

Do we have any test files that are hitting these various encodes/decodes?

cormachallinanderilinx added 3 commits December 19, 2023 17:56

py 2to3 code changes

962e6eb

webassets for python3

69b6517

changing resource to asset

8118d43

cormachallinanderilinx requested a review from EricSoroos December 19, 2023 18:17

EricSoroos requested changes Jan 8, 2024

View reviewed changes

cormachallinanderilinx added 6 commits January 8, 2024 15:51

PR comments - mainly removing list and using unicode_safe

ef3c61a

fixing csv download and upload

a622304

fixing publisher downloads

bd144f1

fixing webassets

8d19d40

js required main js

24cf54a

archiver download encode

36ba9ca

EricSoroos reviewed Jan 9, 2024

View reviewed changes

improving spreadsheet csv upload

04aecfe

cormachallinanderilinx merged commit 67040c4 into master Jan 17, 2024

robredpath mentioned this pull request Feb 1, 2024

Format of the publisher download endpoint has changed #426

Closed

simon-20 mentioned this pull request Feb 1, 2024

Some CSV files have Python's bytes literal marker in the contents of the field #430

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Py 2to3 #420

Py 2to3 #420

cormachallinanderilinx commented Dec 19, 2023

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 8, 2024

EricSoroos Jan 9, 2024

EricSoroos Jan 9, 2024

EricSoroos Jan 9, 2024

	if 'owner_org' in list(data_dict.keys()):
	if 'owner_org' in data_dict:

		@@ -226,7 +226,8 @@ def csv_upload_datasets():
		vars['file_name'] = csv_file.filename
		data = io.BytesIO(_data)

Py 2to3 #420

Py 2to3 #420

Conversation

cormachallinanderilinx commented Dec 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment