Add musuems_victoria_fetch.py by sbarhin · Pull Request #216 · creativecommons/quantifying

sbarhin · 2025-10-29T09:03:42Z

This PR implements automation for Museum Victoria data fetching as discussed in issue #215. The implementation follows the established patterns from existing fetch scripts.
This purpose of this file is to fetch all the records from the Museum Victoria API, then saving the necessary response fields needed for the next phase (processing phase).

Fixes

Related to Add Museum Victoria as data source #215 by @sbarhin

Description

Fetches data for all record types (article, item, specimen, species) from the Museum Victoria API
Prepares and saves meaningful responses into a csv file under the data/2025Q4/1-fetch directory
Next actions will be to process and report the data once the fetching script is approved by reviewers

Checklist

I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
My pull request doesn't include code or content generated with AI.
My pull request has a descriptive title (not a vague title like Update index.md).
My pull request targets the default branch of the repository (main or master).
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no
visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

TimidRobot · 2025-10-30T08:35:55Z

.github/workflows/1-fetch.yml

Please remove these changes from the pull request (PR). Automation will be implemented after this PR is merged.

@sbarhin it is unhelpful to mark a conversation resolved when the pull request (PR) hasn't been updated with a resolution. Please don't.

TimidRobot · 2025-10-30T08:36:57Z

I added or updated tests for the changes I made (if applicable).

I don't see any added or updated tests. Why did you check this item?

I added or updated documentation (if applicable).

I don't see any added or updated documentation. Why did you check this item?

TimidRobot · 2025-10-30T08:41:07Z

scripts/1-fetch/museums_victoria_fetch.py

+            except requests.exceptions.RequestException as e:
+                LOGGER.error(f"Error fetching page {current_page} for {record_type}: {e}")
+                break


Please model exception handling on existing scripts:

quantifying/scripts/1-fetch/github_fetch.py

Lines 152 to 157 in ddb3958

except requests.HTTPError as e:

raise shared.QuantifyingException(f"HTTP Error: {e}", 1)

except requests.RequestException as e:

raise shared.QuantifyingException(f"Request Exception: {e}", 1)

except KeyError as e:

raise shared.QuantifyingException(f"KeyError: {e}", 1)

@sbarhin it is unhelpful to mark a conversation resolved when the pull request (PR) hasn't been updated with a resolution. Please don't.

I apologize. I will take note next time.

TimidRobot · 2025-10-30T08:45:04Z

scripts/1-fetch/museums_fetch.py

Please be sure to run static analysis before committing

Please in the pre-commit-config.yaml, can I update the python version to 3.13?

TimidRobot · 2025-10-30T08:46:13Z

scripts/1-fetch/museums_victoria_fetch.py

+
+def initialize_data_file(file_path, header):
+    if not os.path.isfile(file_path):
+        with open(file_path, "w", newline="") as file_obj:


Please update this per:

Constistent encoding (UTF-8) and newlines (unix) across platorms #217

TimidRobot · 2025-10-30T08:46:22Z

scripts/1-fetch/museums_victoria_fetch.py

+            'MEDIA JSON': sanitize_string(media_json_string),
+        }
+        initialize_data_file(FILE_RECORDS, HEADER_RECORDS)
+        with open(FILE_RECORDS, "a", newline="") as file:


Please update this per:

Constistent encoding (UTF-8) and newlines (unix) across platorms #217

TimidRobot · 2025-10-30T08:53:50Z

scripts/1-fetch/museums_fetch.py

Please rename to match convention.

The script should be executable:

Running the scripts

File permissions

@TimidRobot Hi there, please can you explain further on "renaming to match convention"?

Look at how the fetch scripts are name in the main branch. Match that convention.

Can I name it museums_fetch.py? I guess adding the victoria made it quite longer

TimidRobot · 2025-10-30T08:57:25Z

@sbarhin Please pay better attention to the documentation, instructions, and the conventions set by the existing scripts.

TimidRobot · 2025-10-30T08:59:02Z

scripts/1-fetch/museums_victoria_fetch.py

+        "--enable-save",
+        action="store_true",
+        help="Enable saving results",
+        default="true"


As demonstrated in the existing scripts, this shouldn't default to true.

The scripts should run as safely as possible without any command line options.

@sbarhin it is unhelpful to mark a conversation resolved when the pull request (PR) hasn't been updated with a resolution. Please don't.

TimidRobot · 2025-10-30T08:59:07Z

scripts/1-fetch/museums_victoria_fetch.py

+        "--enable-git",
+        action="store_true",
+        help="Enable git actions (fetch, merge, add, commit, and push)",
+        default="true"


As demonstrated in the existing scripts, this shouldn't default to true.

The scripts should run as safely as possible without any command line options.

@sbarhin it is unhelpful to mark a conversation resolved when the pull request (PR) hasn't been updated with a resolution. Please don't.

TimidRobot · 2025-10-30T09:05:00Z

@sbarhin good work implementing their API

sbarhin · 2025-10-30T14:13:37Z

@TimidRobot Please it seems this PR is closed. What could be the issue?

oree-xx · 2025-10-30T14:17:43Z

@sbarhin It says you closed it?

sbarhin · 2025-10-30T14:19:45Z

@sbarhin It says you closed it?

Oops. Could I have done that mistakenly? I was pushing the changes in relation to the reviews. I later found that it says I closed it

sbarhin · 2025-10-30T14:20:18Z

@oree-xx @TimidRobot Please what's the way forward now

oree-xx · 2025-10-30T14:28:38Z

@sbarhin You would need @TimidRobot to look into it.
I was looking at this tho https://stackoverflow.com/questions/53735216/reopen-a-pull-request-on-github
But I think the maintainer can resolve it. Just wait a little.

sbarhin · 2025-10-30T14:30:15Z

@sbarhin You would need @TimidRobot to look into it. I was looking at this tho https://stackoverflow.com/questions/53735216/reopen-a-pull-request-on-github But I think the maintainer can resolve it. Just wait a little.

Okay, thank you.

sbarhin · 2025-10-30T15:34:15Z

I had issues changing the git credentials. I will create a new PR

sbarhin requested review from a team as code owners October 29, 2025 09:03

sbarhin requested review from TimidRobot and possumbilities and removed request for a team October 29, 2025 09:03

cc-open-source-bot added this to TimidRobot Oct 29, 2025

cc-open-source-bot moved this to In review in TimidRobot Oct 29, 2025

TimidRobot self-assigned this Oct 30, 2025

TimidRobot reviewed Oct 30, 2025

View reviewed changes

updates

6c1cb06

TimidRobot changed the title ~~Added musuems_victoria_fetch.py~~ Add musuems_victoria_fetch.py Oct 30, 2025

lazycoder1996 added 2 commits October 30, 2025 14:05

Reviewed PR changes

1e412c0

Reviewed PR changes

5962205

sbarhin closed this Oct 30, 2025

sbarhin force-pushed the whiz branch from 3206270 to 5962205 Compare October 30, 2025 14:09

github-project-automation bot moved this from In review to Done in TimidRobot Oct 30, 2025

	except requests.HTTPError as e:
	raise shared.QuantifyingException(f"HTTP Error: {e}", 1)
	except requests.RequestException as e:
	raise shared.QuantifyingException(f"Request Exception: {e}", 1)
	except KeyError as e:
	raise shared.QuantifyingException(f"KeyError: {e}", 1)

Uh oh!

Conversation

sbarhin commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes

Description

Checklist

Developer Certificate of Origin

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimidRobot commented Oct 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimidRobot commented Oct 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TimidRobot commented Oct 30, 2025

Uh oh!

sbarhin commented Oct 30, 2025

Uh oh!

oree-xx commented Oct 30, 2025

Uh oh!

sbarhin commented Oct 30, 2025

Uh oh!

sbarhin commented Oct 30, 2025

Uh oh!

oree-xx commented Oct 30, 2025

Uh oh!

sbarhin commented Oct 30, 2025

Uh oh!

sbarhin commented Oct 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sbarhin commented Oct 29, 2025 •

edited

Loading