Encoded image strings /w gpt-4-vision-preview

[antoan]

Is it currently possible to pass encoded image strings to Marvin, to extract structured data, rather than external urls?

Any thoughts would be much appreciated.

[zzstoatzz]

hi @antoan this should be possible as of #834 or v2.2.2 - does that seem like what you need?

example:

In [1]: import marvin

In [2]: link = "https://plus.unsplash.com/premium_photo-1674917000586-b7564f21540e?q=80&w=4912&auto=format&fit=crop&ixlib=rb-
   ...: 4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D"

In [3]: import httpx
   ...:
   ...: marvin.beta.caption(marvin.beta.Image(data=httpx.get(link).content))
Out[3]: 'A serene sunrise over a mountainous landscape with a winding road and alpine wildflowers in the foreground. The mountain peak is partly veiled by light morning mist.'

here im getting bytes from the image (which happens to live at an external url, just because that's easier to share), which you could swap for your own image's bytes

[zzstoatzz]

closing this, please let me know if 834 does not solve your issue