Allow configuring timeout setting on sqlpage.fetch

[djyotta]

What are you building with SQLPage ?

Personal tools too share copy-pasted text, long urls (to long to type if that's the only option, ie over the phone), files.

What is your problem ? A description of the problem, not the solution you are proposing.

Uploaded files can be quite large in my use case. I want to support ~100MB and even ~1GB. Typically, files will be small (~2MB photos), but if there is the one off large file, I want to support that.

Anyhow, a 100MB file is large enogh that when I use sqlpage.fetch to get the file hash from an external service, the time it takes the service to hash the file is longer than the default 5 seconds timeout.

What are you currently doing ? Since your solution is not implemented in SQLPage currently, what are you doing instead ?

I have no work around.

Files larger than ~65MB may not get processed correcly.

Describe the solution you'd like

I'd like to expose the request timeout in sqlpage.fetch:
https://docs.rs/awc/latest/awc/struct.ClientBuilder.html#method.timeout

I think the default of 5 seconds is good enough, but abilty to override once off for known slow responses would be good.

A configurable default would be good too for cases where bandwidth is very limited.

A clear and concise description of what you want to happen.

Describe alternatives you've considered

I considered using sqlpage.exec to calculate the hash, but would rather not use that method due to security concerns.

[lovasoa]

Hello! I agree with the need for configuring the timeout. Do you want to make a pull request?

Independently of the above, What security concerns do you have with sqlpage.exec ? It sounds like a better way to compute a file hash.

[djyotta]

RE: sqlpage.exec
Nothing in particular (other than the obvious). I just didn't want to enable it if I didn't absolutely have to.
In this particular case, it should be totally safe as the only program argument will be the file path which (I assume) can not be set by the user.

I do use it in an app that is not public facing so somewhat less risk of abuse - in that one it's particularly dangerous too because I build the command line from db content (yikes).

I'm thinking about separating my three utilities clip/got/upload into three separate apps - then I could enable sqlpage.exec on the upload one alone - I'd be fine with that.

Still not sure about the hash calculation being done in a script - all the other tools use the url (I'm using a blake2 hash with message authentication, so not a simple hash the OS can provide). I could use curl (or other) in my script to avoid code duplication, but it just feels janky...

[lovasoa]

I think sqlpage.exec('my_hash.sh', sqlpage.uploaded_file_path('myfile')) would be better for this use case...
Even sqlpage.exec('my_hash.sh', $user_controlled) is safe. The user cannot control more than the single argument to the script; this does not give arbitrary execution power to the user.

[djyotta]

Well maybe not so safe on windows... https://blog.rust-lang.org/2024/04/09/cve-2024-24576.html

And while Linux isnt affected by that, if that arg is used unquoted anwhere in a bash script it could be disastrous.

But in my case, my hashing is done in a python script. So you have almost convinced me. I do have the extra headache of running in docker, so I would have to wrap your image with the extra dependencies... not so fun especially with python. Alternatively, I could build a binary with pyinstaller and mount it via bind mount (assuming the image has glibc)... That would be a more portable solution in any case... I will sleep on it.

[djyotta]

@lovasoa The timeout feature is working great. I've set it to 15000 ms and this is long enough to hash a 128MB file.

[lovasoa]

Here is an example of a Dockerfile setup that allows you to easily add external software to run together with sqlpage: #123 (comment)

[djyotta]

Here is an example of a Dockerfile setup that allows you to easily add external software to run together with sqlpage: #123 (comment)

This is helpul thanks. It makes way more sense to copy sqlpage binary into my python docker image.