Skip to content

Upload Subject Attached Media#331

Merged
yuenmichelle1 merged 18 commits intomasterfrom
ancillary-data-upload
Nov 3, 2025
Merged

Upload Subject Attached Media#331
yuenmichelle1 merged 18 commits intomasterfrom
ancillary-data-upload

Conversation

@yuenmichelle1
Copy link
Collaborator

@yuenmichelle1 yuenmichelle1 commented Oct 6, 2025

Upload Subject attached media/ attached images/ancillary data for existing subjects

Example usages:

Without parallelism and multiple threads

subject = Subject(1234)
local_files = [...]

for file_location in local_files: 
  future_result = subject.save_attached_image(file_location)
  try:
    future_result.result()
  except Exception e:
    print(f"Upload failed for {file_location}")
    print(e)
   

Utilizing parallelism

from concurrent.futures import as_completed

subject = Subject(1234)
local_files = [...]

with Subject.async_saves():
    future_to_file = {subject.save_attached_image(file_location): file_location for file_location in local_files}
    for future in as_completed(future_to_file):
        local_file = future_to_file[future]
        try:
            future.result()
        except Exception as exc:
            print(f"Upload failed for {local_file}")

@yuenmichelle1 yuenmichelle1 marked this pull request as ready for review October 7, 2025 00:41
@yuenmichelle1 yuenmichelle1 requested a review from lcjohnso October 7, 2025 00:41
@yuenmichelle1
Copy link
Collaborator Author

@lcjohnso this is ready for review. Do you mind reading through the doc strings and examples to make sure wording makes sense?

Bonus: if you can run the examples and see what extra info is needed that I did not include in docs.

Maybe worth pinging @astrohayley after an initial run through from Cliff.

Copy link
Member

@lcjohnso lcjohnso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest a number of docs edits, and I have questions about how users are meant to use the new functions. It appears save_attached_image() is the primary interface, but the availability of add_attached_image() clouds this assumption.

- **metadata** optional dict, serves as metadata for the attached_image/media
- **client** optional Panoptes.client() instance. Sent as a parameter for threading purposes for parallelization so that thread uses the correct client context.

Examples::
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add in-line comments (with # mark)

yuenmichelle1 and others added 4 commits October 31, 2025 12:24
Co-authored-by: Cliff Johnson <cliff@zooniverse.org>
Co-authored-by: Cliff Johnson <cliff@zooniverse.org>
Co-authored-by: Cliff Johnson <cliff@zooniverse.org>
Co-authored-by: Cliff Johnson <cliff@zooniverse.org>
@PmasonFF
Copy link

PmasonFF commented Oct 31, 2025

At one point you are stating:

 - **attached_media** can be an open :py:class:`file` object, a path to a
+          local file, or a :py:class:`dict` containing MIME types and URLs for
+          remote media.

This is no longer quite correct if the user has not installed libmagic and mime type detection is being handled by mimetypes - mimetypes does not handle all types of :py:class:file object, specifically BytesIO() file objects work with libmagic but not mimetypes.... This means one can no longer process an image (compress or crop, add captions etc) in memory and add the BytesIO() object as a attached image or location.

@lcjohnso
Copy link
Member

This is no longer quite correct if the user has not installed libmagic and mime type detection is being handled by mimetypes - mimetypes does not handle all types of :py:class:file object, specifically BytesIO() file objects work with libmagic but not mimetypes.... This means one can no longer process an image (compress or crop, add captions etc) in memory and add the BytesIO() object as a attached image or location.

Hi @PmasonFF -- For the case where libmagic is not installed/available and the user wants to pass a BytesIO object, one can override and avoid any form of type sensing by using the manual_mimetype parameter.

…ched_image a private method since used internally, add eg. on saved attached image for externally hosted files
@yuenmichelle1 yuenmichelle1 merged commit 78b1cea into master Nov 3, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants