server: add support for local image path loading for server

[cchadowitz]

image_url now supports file:// URIs to load images from local file paths. Similar to vLLM, this is guarded by a new parameter --allowed-local-media-path (default none), as well as a maximum file size parameter --local-media-max-size-mb (default 15mb).

[cchadowitz]

I also locally ran bash ./ci/run.sh ./tmp/results ./tmp/mnt and encountered no errors/failures.

[ngxson]

IMO I don't feel like adding this feature as it can be a huge security risk. Better to do it only when you can list out all of the edge cases and do proper testing.

[ngxson]

can't we simply do string_replace_all(url, "file://", "") ?

[cchadowitz]

Instead of url.substr(7)? I thought strictly stripping off the file:// prefix was the minimal modification necessary to the potential file path prior to checking and validating it was legitimate. E.g. if for some reason it was a malformed path with file:// later on, this would also replace that. Maybe that doesn't matter since it would already be an invalid path but it could introduce a case where it leads to an unexpectedly valid path by removing this in the middle.

[ngxson]

maybe better using std::ifstream

[cchadowitz]

Happy to, but I was trying to take cues from how this was done in https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/mtmd-helper.cpp#L439

[ngxson]

opt.allowed_local_media_path = "/home/user";
fname = "/home/userabc"
// string_starts_with(fname, opt.allowed_local_media_path) can be true right?

[cchadowitz]

Good point - in theory this would only be an issue if both /home/user and /home/userabc are valid directories, in which case it's somewhat ambiguous if setting opt.allowed_local_media_path = "/home/user" is intended to allow both dirs or strictly /home/user, but I would argue it should only allow the exact dir specified. I'll consider how best to account for this.

[cchadowitz]

IMO I don't feel like adding this feature as it can be a huge security risk. Better to do it only when you can list out all of the edge cases and do proper testing.

This is my first time looking at or contributing to llama.cpp - can you point me in the right direction for where/how to add test cases in the codebase for this type of feature? I'm happy to do so, just a bit lost.

I was trying to copy the style and approach as much as possible from other parts of the existing llama.cpp codebase (e.g. fopen instead of std::ifstream), but if that's not necessary, I can certainly use std::filesystem to better handle paths and edge cases in a safer (and more maintainable) way.

[cchadowitz]

I've swapped to using std::ifstream and std::filesystem, making it cleaner and more secure. I also validate the parameter on start up instead of at the time of processing a request, and use std::filesystem::file_size to avoid opening the file until after the max filesize check.

I also added test cases to tools/server/tests/unit/test_vision_api.py to exercise the two new arguments. I wasn't sure if something needed to be added to tests/test-arg-parser.cpp for the new args, or if they will be tested automatically as part of the existing test suite there.

Please let me know if there are additional tests you'd like me to add (either for the arg parser, or the test_vision_api.py suite).