Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add custom RAW format conversion tool support #929

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions Dockerfile
Copy link
Contributor

@kkovaletp kkovaletp Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RUN if [ "${TARGETPLATFORM}" = "linux/amd64" ] || [ "${TARGETPLATFORM}" = "linux/arm64" ]; then \
  apt install -y darktable; fi

As we add the support of ImageMagick, it would be great to enhance this by adding the else with ImageMagick installation. Also, it makes sense to move just created 3 ENV lines to the then branch of this if, setting the same 3 variables in the else to the values, compatible with the ImageMagick

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ImageMagick is not in conflict with Darktable, we can keep it the way it is, i.e. Darktable is installed by default, and other possible handlers are advanced usage. Because I can't be sure which environments Darktable can't be installed and ImageMagick can, in the current issues, there is no known problem of not being able to handle RAW formats because darktable can't be installed!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no known problem of not being able to handle RAW formats because Darktable can't be installed!

Actually, as you can see from the mentioned if, Darktable doesn't support (and cannot be installed on) ARM-based systems, some right now Photoview, deployed on ARM doesn't support RAW formats at all.

I cannot test it, as I don't have such a system, but as far as I know, ImageMagick supports ARM and can be installed there - that is why I've proposed this change: to finally extend the RAW support to the ARM platform as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkovaletp I think support for arm based images and using raw can be added at a later date under another issue so we can get this piece in. I think OP has said in the past he doesn't use docker, so unless he's willing to do it we should make another issue for it so someone with more expertise can implement it correctly.

@fierceX It looks like you've pulled in a load of extra commits somehow, bad rebase maybe? when you fix those up I'll give a proper review but a quick glance through it looks good. The only thing missing looks like defaults when running without docker...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jordy2254 I merged in commits that were already merged into master, and now how do I cancel those commits in order to prevent my changes from conflicting with some existing changes?

Copy link
Member

@jordy2254 jordy2254 Apr 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fierceX general process would be;

  • sync your fork's master branch with photoview
  • switch the master branch on your dev machine do a git pull
  • switch back to your feature branch
  • this is where things can differ personally I use the first
    • git rebase master (you can do -i if you'd like to change commitsor
    • OR git merge master will do the job
  • Fix the conflicts you get from either method. a rebase -i will let you manually drop those stray commits you seemed to of picked up somehow.

If you need more support feel free to reach out.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jordy2254 I pushed a new commit and it should now work fine, the change file contains only the changes needed for this PR。Other than that, do I need to adjust anything?

Copy link
Member

@jordy2254 jordy2254 Apr 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fierceX sorry for the late reply commit history and file changes look fine now thanks :)
I'm not on the computer for the next few days but spotted one minor thing I've commented on.

By any chance have you looked at the test failures on your branch yet? Do you know what's up with them?

Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,9 @@ ENV PHOTOVIEW_LISTEN_PORT 80

ENV PHOTOVIEW_SERVE_UI 1
ENV PHOTOVIEW_UI_PATH /ui
ENV PHOTOVIEW_RAW_PROCESSING_EXECUTABLE="darktable-cli"
ENV PHOTOVIEW_RAW_PROCESSING_EXECUTABLE_CHECK="--version"
ENV PHOTOVIEW_RAW_PROCESSING_ARGS="{{ .InputPath }} {{ .OutputPath }} --core --conf plugins/imageio/format/jpeg/quality={{ .Quality }} --configdir /tmp/photoview-convert"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this lead to a command injection attack, by crafting a malicious environment variable?

And if so is it a problem?

Just wanted to mention it to hear what you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@viktorstrate, This is a great catch!
Yes, from a security point of view, this is a critical vulnerability, as we cannot know in advance which executable should be called by the Photoview and rely on the external config (environment variable in this case), which can be modified.
To mitigate this we need to statically resolve the path to the executable at the build time and accept later only the commandline parameters from the variables. This also means that we will need to preinstall all supported executables inside the same image and define paths to them in Dockerfile providing the ability to choose an executable from the predefined list in the scanner options on the Settings page. A similar approach might work for a directly hosted app, where all paths should be provided in some form at the build time, so they are available on the Settings later. If a user wants to add a custom executable (a self-developed one or just something, not tested by us and not supported from the project perspective) in the case of deployment with docker, a local rebuild of the Photoview image will be required.
An alternative way would be to define the AppArmor profile for the Photoview image and allow only a predefined list of executables to be executed inside the container. However, this is more difficult to maintain and would require strong Linux admin skills for a user in the case of direct deployment to a host instead of using docker (which, I believe, is a stop-factor for us, as our target audience is not only Linux gurus but also regular users), while doesn't provide much more flexibility, as we just shift the place of executables hardcoding from the code to the AppArmor profile, so to add a custom executable, the profile should be changed, which requires an image rebuild as well.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this issue is within the scope of consideration. The prerequisite for attacking through custom environment variables is that the attacker has obtained the permissions of the machine. If the attacker has obtained the permissions of the machine, then the security issue does not require us to consider it. In addition, regarding the docker image, the program and environment variables used for RAW conversion have been pre-installed in the official image. If the program or the value can be artificially modified, the attacker still needs to obtain the permissions of the machine. From what perspective, this should not be an issue that ordinary applications need to consider.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An attacker might get access to the docker container and put there a new malicious executable, then manipulate with env variables to replace the RAW processing one, so that the malicious executable is called instead. In this case, there is no need to have access to the host.
If Photoview is deployed on the host directly, this is the same system, so an attacker has access to the host, as a 1st step.
But in both cases, there are different levels of access: to put a file to the home folder and modify an env variable you don't need admin permissions. The photoview user can do that. While to modify an existing executable in the system (owned by root), which is registered in the Photoview app, an attacker should become an admin 1st.
Please see the PR #863, where I implemented many security enhancements for our docker image, so now the Photoview app is executed there under non-root user, which has write access to his home folder and mounted app cache (which by default is also inside the home folder). All executables remain owned by root and cannot be modified without root permissions.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If an attacker can modify environment variables and restart photoview, he also has permission to directly execute the malicious program. In addition, environment variables can be changed, so even under existing circumstances, the default RAW converter and video converter will still be replaced without requiring root permission. Just modify the PATH environment variable

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, environment variables can be changed, so even under existing circumstances, the default RAW converter and video converter will still be replaced without requiring root permission. Just modify the PATH environment variable

I really like this argument, this PR would not make Photoview any less secure because of this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, we need to prepare an AppArmor profile, which will at least forbid the execution of any apps from non-standard locations, or allow the execution of only allowed binaries - I need to think about this more...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and video converter

BTW, if we provide a way to register any external RAW processing app, do we want to provide the same approach for video-converter integration as well?
How does the scanner create thumbnails for non-RAW images: does it use the same RAW-processing app integration? if not, probably, we need to change that as well? I assume that there is a low probability that a user wants to use a non-default app to process RAW files while keeping the standard one for regular images.
@viktorstrate, @jordy2254, @fierceX, what do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the first message I don't think it's something that we need to do unless anyone raises a concern. Ultimatly the attack vector is minimised in a docker environment and on a bare metal install I'd expect people to manage it as they see fit based on there use case.
For the latter message In my opinion out of scope for this piece of work. lets keep things in scope and make additional issues/PRs for them as needed to try and keep risk to a minimum and follow CI/CD better.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the latter message In my opinion out of scope for this piece of work

If you refer to the "do we want to provide the same approach for video-converter integration as well?" message, sure, I agree that this might be a new PR. My question was just from a product functionality perspective: do you see the inconsistency between the flexibility of the RAW processing app integration and video / thumbnails converters, possibly some other workers?
This is a good approach to give the user an opportunity to redefine what a 3rd-party app should process the incoming media. I'd like to have it available for all types of workers at the final stage (after all corresponding PRs are merged)


EXPOSE 80

Expand Down
65 changes: 37 additions & 28 deletions api/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,52 +3,61 @@ module github.com/photoview/photoview/api
go 1.18

require (
github.com/99designs/gqlgen v0.17.12
github.com/99designs/gqlgen v0.17.45
github.com/Kagami/go-face v0.0.0-20210630145111-0c14797b4d0e
github.com/barasher/go-exiftool v1.8.0
github.com/barasher/go-exiftool v1.10.0
github.com/buckket/go-blurhash v1.1.0
github.com/disintegration/imaging v1.6.2
github.com/go-sql-driver/mysql v1.6.0
github.com/gorilla/handlers v1.5.1
github.com/gorilla/mux v1.8.0
github.com/gorilla/websocket v1.5.0
github.com/go-sql-driver/mysql v1.8.1
github.com/gorilla/handlers v1.5.2
github.com/gorilla/mux v1.8.1
github.com/gorilla/websocket v1.5.1
github.com/h2non/filetype v1.1.3
github.com/joho/godotenv v1.4.0
github.com/joho/godotenv v1.5.1
github.com/otiai10/copy v1.7.0
github.com/pkg/errors v0.9.1
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06
github.com/stretchr/testify v1.8.0
github.com/strukturag/libheif v1.12.0
github.com/vektah/gqlparser/v2 v2.4.6
github.com/stretchr/testify v1.9.0
github.com/strukturag/libheif v1.15.1
github.com/vektah/gqlparser/v2 v2.5.11
github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0
github.com/xor-gate/goexif2 v1.1.0
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d
golang.org/x/image v0.0.0-20220617043117-41969df76e82
gopkg.in/vansante/go-ffprobe.v2 v2.0.3
gorm.io/driver/mysql v1.3.4
gorm.io/driver/postgres v1.3.8
gorm.io/driver/sqlite v1.3.5
gorm.io/gorm v1.23.7
golang.org/x/crypto v0.22.0
golang.org/x/image v0.15.0
gopkg.in/vansante/go-ffprobe.v2 v2.1.1
gorm.io/driver/mysql v1.5.6
gorm.io/driver/postgres v1.5.7
gorm.io/driver/sqlite v1.5.5
gorm.io/gorm v1.25.9
)

require (
filippo.io/edwards25519 v1.1.0 // indirect
github.com/agnivade/levenshtein v1.1.1 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/felixge/httpsnoop v1.0.3 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgconn v1.12.1 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20200714003250-2b9c44734f2b // indirect
github.com/jackc/pgtype v1.11.0 // indirect
github.com/jackc/pgx/v4 v4.16.1 // indirect
github.com/jackc/pgservicefile v0.0.0-20231201235250-de7065d80cb9 // indirect
github.com/jackc/pgx/v5 v5.5.5 // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/mattn/go-sqlite3 v1.14.14 // indirect
github.com/kr/text v0.1.0 // indirect
github.com/mattn/go-sqlite3 v1.14.22 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
golang.org/x/text v0.3.7 // indirect
github.com/rogpeppe/go-internal v1.12.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sosodev/duration v1.2.0 // indirect
github.com/urfave/cli/v2 v2.27.1 // indirect
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
golang.org/x/mod v0.16.0 // indirect
golang.org/x/net v0.24.0 // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.19.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
319 changes: 77 additions & 242 deletions api/go.sum

Large diffs are not rendered by default.