Additional initializer #23

Andrey170170 · 2025-02-04T15:22:39Z

Added new initializers - fathom_net, EoL and Lila

Small fixes:

added MATERIAL_CITATION filtering for gbif initializer, issue described here
formatting updates
consolidate dependencies in pyproject
adds descriptions of downloaded data format

Small fixes Also adjusted tools to use `source_id` instead of `gbif_id`

Added tool_name_override option for Tools, to be able to use custom tools

Added the use of `verification_scheme` instead of hard coded column names for some of the parts of the runner

plus some minor fixes

Updated readme - added `how to access data` section. Updated pyproject.toml - added dependency libraries directly into this file, instead of link to `requirements.txt`.

Now there is a distinction between scheduled filtering or scheduling jobs and completed ones. Adjusted logic of scripts according to this change.

Extracted initializers into a class structure Rewrote initialization calling file to have a dict with initialization types. Added `initializer_type` in mandatory config fields

Small fix

egrace479 · 2025-02-06T15:45:11Z

Please add description of base initializer, inheritance to child initializers, and considerations for making a custom child initializer. Use existing as examples, e.g., filters that could be applied (GBIF excluding MATERIAL_CITATION). This would be a good place to note the importance of understanding the metadata coming from the source before creating a child initializer and to not rely on source IDs to be persistent (also check uniqueness if relying on them to map to additional metadata)---considering EOL content IDs, which may be unique but it's the page ID that maps to the taxa information.

As discussed, put this description into a README in the initializer/ folder and link to it from the root repo README.

Added README.md to initializer. Made small code quality adjustments to initializers

Added doc strings to `base_initializer.py`

thompsonmj · 2025-04-03T14:18:27Z

Todo: add comments to base initializer and individual data sources as well.

README.md

egrace479

A few more suggestions and questions.

src/distributed_downloader/core/initializers/README.md

egrace479 · 2025-04-08T14:04:33Z

src/distributed_downloader/core/initializers/README.md

+  And lastly, it removes any entries that have `MATERIAL_CITATION` in `basisOfRecord`.
+- `FathomNetInitializer`: Initializer for the FathomNet dataset. It filters out any entries without an `uuid` or `url`
+  value.
+  Additionally, removes any entries that are "not valid" by the `valid` column.


Do you recall what determined if an entry was valid in FathomNet?

src/distributed_downloader/core/initializers/README.md

src/distributed_downloader/core/initializers/eol_initializer.py

src/distributed_downloader/core/initializers/README.md

pyproject.toml

src/distributed_downloader/core/initializers/README.md

Added docstrings to python files Adjusted main `README` Deleted unused `multimedia_scheme` file

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Updated package requirements

README.md

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

src/distributed_downloader/core/fake_profiler.py

egrace479

Suggested generalization for clarifying uuids (beyond just TreeOfLife).

README.md

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Updated dependency list Updated readmes

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

src/distributed_downloader/core/initializers/README.md

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Fix in checkpoint

src/distributed_downloader/core/mpi_downloader/dataclasses.py

egrace479 · 2025-04-16T20:14:15Z

src/distributed_downloader/core/mpi_downloader/dataclasses.py

+        competed_queue: Queue of completed batches
+        total_batches: Total number of batches to process
+        done_batches: Number of batches that have been processed
+    """
+
    server_name: str
    download_complete: threading.Event
    competed_queue: queue.Queue[CompletedBatch]


There is a typo here: "competed_queue" instead of "completed_queue" It seems to only exist here (search result), so should be a simple fix.

dataclasses.py misspelling fix

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Andrey170170 and others added 10 commits August 7, 2024 21:04

Fathom_net initializer

1e18716

Fathom_net update

84adde8

Small fixes Also adjusted tools to use `source_id` instead of `gbif_id`

Small fixes

95e8bef

Added tool_name_override option for Tools, to be able to use custom tools

Small fixes

ddad5f2

Added the use of `verification_scheme` instead of hard coded column names for some of the parts of the runner

EoL download

bff688f

plus some minor fixes

ReadMe file update

0edeeb8

Updated readme - added `how to access data` section. Updated pyproject.toml - added dependency libraries directly into this file, instead of link to `requirements.txt`.

Update README.md

049e086

Lila download

035839e

Updated tools checkpoint options

08f7a62

Now there is a distinction between scheduled filtering or scheduling jobs and completed ones. Adjusted logic of scripts according to this change.

Added filtration to gbif initializer using basisOfRecords

bbe77c4

egrace479 added documentation Improvements or additions to documentation enhancement New feature or request labels Feb 4, 2025

Andrey170170 added 2 commits February 6, 2025 01:54

Initializers refactoring

34ecfd7

Extracted initializers into a class structure Rewrote initialization calling file to have a dict with initialization types. Added `initializer_type` in mandatory config fields

Initializers refactoring

307054b

Small fix

Andrey170170 added 2 commits February 9, 2025 23:47

Documentation

03b265f

Added README.md to initializer. Made small code quality adjustments to initializers

Documentation

b37de48

Added doc strings to `base_initializer.py`

Andrey170170 requested review from egrace479 and thompsonmj February 14, 2025 15:26

egrace479 linked an issue Apr 3, 2025 that may be closed by this pull request

Consolidate initializers into Initializer class #24

Closed

egrace479 reviewed Apr 3, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

egrace479 reviewed Apr 8, 2025

View reviewed changes

Update pyproject.toml

91b936c

egrace479 reviewed Apr 10, 2025

View reviewed changes

src/distributed_downloader/core/initializers/README.md Show resolved Hide resolved

Andrey170170 and others added 3 commits April 10, 2025 23:50

Documentation

49601ab

Added docstrings to python files Adjusted main `README` Deleted unused `multimedia_scheme` file

Apply suggestions from code review

56a01d6

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Documentation

0e42a1d

Updated package requirements

egrace479 reviewed Apr 14, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

Update README.md

f033f96

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

egrace479 reviewed Apr 15, 2025

View reviewed changes

src/distributed_downloader/core/fake_profiler.py Outdated Show resolved Hide resolved

egrace479 reviewed Apr 15, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

egrace479 linked an issue Apr 15, 2025 that may be closed by this pull request

Clean up requirements to avoid listing extra dependencies #6

Closed

egrace479 mentioned this pull request Apr 15, 2025

Set up PyPI workflow #30

Merged

Andrey170170 and others added 3 commits April 16, 2025 13:41

Apply suggestions from code review

6f2d463

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Small fixes

ffe80da

Updated dependency list Updated readmes

Update src/distributed_downloader/core/fake_profiler.py

c344b1b

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

egrace479 reviewed Apr 16, 2025

View reviewed changes

src/distributed_downloader/core/initializers/README.md Outdated Show resolved Hide resolved

Andrey170170 and others added 2 commits April 16, 2025 14:20

Update src/distributed_downloader/core/initializers/README.md

d4b05c0

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Small fixes

44d8a12

Fix in checkpoint

egrace479 reviewed Apr 16, 2025

View reviewed changes

src/distributed_downloader/core/mpi_downloader/dataclasses.py Outdated Show resolved Hide resolved

egrace479 reviewed Apr 16, 2025

View reviewed changes

src/distributed_downloader/core/mpi_downloader/dataclasses.py Outdated Show resolved Hide resolved

egrace479 reviewed Apr 16, 2025

View reviewed changes

Andrey170170 and others added 2 commits April 16, 2025 23:41

Small fixes

944ba4b

dataclasses.py misspelling fix

Apply suggestions from code review

9a1f145

Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>

Andrey170170 requested a review from egrace479 April 17, 2025 03:47

egrace479 approved these changes Apr 17, 2025

View reviewed changes

Andrey170170 merged commit a4a94f3 into main Apr 17, 2025

Andrey170170 deleted the eol_download branch April 17, 2025 13:49

Additional initializer #23

Additional initializer #23

Uh oh!

Conversation

Andrey170170 commented Feb 4, 2025 • edited by egrace479 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

egrace479 commented Feb 6, 2025

Uh oh!

thompsonmj commented Apr 3, 2025

Uh oh!

Uh oh!

egrace479 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

egrace479 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egrace479 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

egrace479 Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Andrey170170 commented Feb 4, 2025 •

edited by egrace479

Loading