use URL constructor to handle all types of links

[ape-nq]

This should enable to handle all types of relative URLs defined in RFC: https://www.rfc-editor.org/rfc/rfc3986#section-5.4

[HenryHengZJ]

hmm can you give example where the particular link cant be scraped?

[ape-nq]

Example is in the issue.

I added screenshots for Release and patched version:
#1741 (comment)

[ape-nq]

I also added a Chatflow to reproduce the example:

• https://github.com/FlowiseAI/Flowise/files/14310194/Relative.Link.Repro.Chatflow.json

[HenryHengZJ]

@chungyau97 can you take a look at this?

[chungyau97]

Hi @ape-nq, thanks for the PR.

I wasn't aware that URL() will handle href and baseurl itself.
https://developer.mozilla.org/en-US/docs/Web/API/URL/URL#examples

[chungyau97]

Hi @ape-nq, you need to update your repo main branch and merge it into this branch to resolve run fail.

[ape-nq]

You're welcome!

Thank you and the other Flowise contributors! It really helped me to get started with RAG chatflows. Glad I could help by sending my fix upstream.