Make sure the URL to capture is actually a URL

[Rafiot]

          Hello there,

I would like to report a similar "bug".

I recently discovered in one of my workflow, that I was submitting bogus URL to pylookyloo, by prefixing a single-quote by a double-quote, and pylookyloo accepted it anyway, leading to bogus scan.

Example of my bogus workflow :

$ lookyloo --listing --query "'http://google.fr'"
https://lookyloo.circl.lu/tree/18f658c0-48b3-47ae-8b5d-35b3bd4c7fc1

But if you look at the lookyloo scan, it does not work as the URL was not correctly formatted.

Hence, could you improve the URL argument parsing in order to prevent bogus results ?
For instance by removing all potential quotes around the URL, and also by ensuring that the URL is correctly structured (using this lib ?)

Cheers, and keep up the good work, CIRCL tools are awesome !

Originally posted by @maaaaz in #4 (comment)

[Rafiot]

@maaaaz I just moved you issue here because the one you were referring to was definitely closed (it was simply to make sure the URL we submit the capture to starts with https://).

In your case, you want to make sure the thing submitted to Lookyloo is a valid URL.

I could add a few checks and stripping characters, but I feel it will quickly become unmanageable as people will pass things to capture in in all kind of weird ways, and making the difference with sane but unhelpful encoding as you have with the quotes and clearly broken and insane will be hard.

Important to also keep in mind that the query passed in the API can be a path on the filesystem, and not necessarily a URL.

It would make more sense to the preprocessing on your side (as you know the data you're getting), and make sure what you pass to lookyloo is valid. What I can do on pylookyloo side is making sure to raise an exception of the query is improperly formatted.

[maaaaz]

What I can do on pylookyloo side is making sure to raise an exception of the query is improperly formatted.

Sure, it would be the best way to fix: from the server-side, return an error when bogus data is sent.

Cheers!

[Rafiot]

Okay, I investigated further, and the fix will be completely server side with no validation from the client. The reason is that you can pass a "google.fr" and lookyloo will take care of adding http:// so validating a URL in pylookyloo will raise exceptions that aren't really a problem.

I'll instead improve the error message and return it to the user.

[Rafiot]

Improve error message: ail-project/LacusCore@dfeb6d6