[DEVX-828] Added image summarization in multimodal pipeline #31
Conversation
| resp = self.model.predict(img_inputs) | ||
|
|
||
| new_elements = [] | ||
| for i, element in enumerate(resp.outputs): | ||
| summary = "" | ||
| if image_elements[i].text: |
There was a problem hiding this comment.
are we sure that the index of image in image_elements is same as in resp.outputs?
| image_data = meta.pop('image_base64', None) | ||
| id = meta.get('input_id', None) |
There was a problem hiding this comment.
QQ: why are we adding this new field ID and will it be used?
There was a problem hiding this comment.
Yes, this is to identify the corresponding summary that was generated for the images in the PDF!
| new_elements = [] | ||
| for i, element in enumerate(resp.outputs): | ||
| summary = "" | ||
| if image_elements[i].text: |
There was a problem hiding this comment.
I believe image elements will not have text, so why this check here?
There was a problem hiding this comment.
I observed that some image elements had text too... it can be seen in the output of 9th cell in this notebook
| """ Summarizes image elements. """ | ||
|
|
||
| def __init__(self, | ||
| pat, |
There was a problem hiding this comment.
PAT can be optional, if not passed, the Clarifai SDK itself will check in env and return a error if is not set in env.
https://clarifai.atlassian.net/browse/DEVX-828