Feat: images in fetched RAG website documents #2658
alexfromapex
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Is your feature request related to a problem? Please describe.
When using
#to load a website as a document, it should pull the full HTML of the website (or be configurable to download images as separate documents, etc). Especially with the rise of multi-modal models like llava this will allow more complex and extremely useful behaviors. E.g. I want to pull the HTML for this website about math problems and then have the model replace the<img />tags with the content of theiraltattributes which contains the LaTeX definition that the image is representing.Describe the solution you'd like
When using
#to fetch websites as documents, the full HTML for the website URL should be pulled, and/or maybe the images could be fetched as separate documents (with a setting).Describe alternatives you've considered
I've tried manually copying and pasting the HTML into the chat but the context length is too short and the models aren't picking up on anything except the last few bits of HTML.
Additional context
This website
https://artofproblemsolving.com/wiki/index.php/2024_AIME_I_Problemscontains some math problems. I'd like to fetch the HTML and then parse out the math problems (including the LaTeX in the imagealtattributes):<img src="//latex.artofproblemsolving.com/a/9/e/a9e826e68f4134acde4bc1d430a580e0e3649cff.png" class="latex" alt="$s+\frac12$" style="vertical-align: -13px" width="46" height="38">Which the model could then extract the LaTeX portion:
$s+\frac12$There's probably a lot of clever ways this could be done but it's a really useful and interesting use-case.
Beta Was this translation helpful? Give feedback.
All reactions