Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch process from command line #2008

Closed
borconi opened this issue Jan 17, 2019 · 6 comments · Fixed by #6054
Closed

Batch process from command line #2008

borconi opened this issue Jan 17, 2019 · 6 comments · Fixed by #6054
Labels
no-issue-activity

Comments

@borconi
Copy link

borconi commented Jan 17, 2019

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I have to process a folder with a huge amount of image and apply a given XMP to all of them. Currently I need create a for loop and call darktable-cli for every image separately.

Describe the solution you'd like
A clear and concise description of what you want to happen.

  • Allow darktable-cli to accept input list, something similar to Ghostscript @filename will work, of course this will mean to specify an output path and extension rather then a filename.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Or give an options for darktable to be run as a daemon, where we can send the input and it will return the output. While a for loop will still be necessary, there will be a significant performance gain because darktable won't need to be reloaded on every iteration.
This will be even more efficient when working with servers.

Additional context
Add any other context or screenshots about the feature request here.

I'm planning to use this as a backend of a photo-sharing platform where we upload over 1 million photos every and applying instagram-like filters on the fly is one of the core functionalities I'm looking to provide.
@PetrGlad
Copy link

Some of documentation (e.g. https://www.darktable.org/usermanual/en/overview_chapter.html#darktable_cli_commandline_parameters) implies that source folder can be a directory. So one could process a bunch of files by grouping them into a separate folder.

I tried this with darktable-cli 2.4.4 and it fails with program error:

darktable-cli --width 3000 --height 3000 --hq true print 'print-exported/$(FILE_NAME).jpg'
[New LWP 21461]
[New LWP 21462]
[New LWP 21504]
[New LWP 21505]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f3eb66473ea in __GI___waitpid (pid=21506, stat_loc=0x0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
30	../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
warning: Currently logging to /tmp/darktable_bt_N5YAWZ.txt.  Turn the logging off and on to make the new setting effective.
#0  0x00007f3eb66473ea in __GI___waitpid (pid=21506, stat_loc=0x0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007f3eb70be150 in  () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#2  0x00007f3eb65aa100 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f3eb70ddd7b in dt_control_progress_create () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#4  0x00007f3eb70d6bd7 in dt_control_job_add_progress () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#5  0x00007f3eb70dd7eb in dt_film_import1_create () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#6  0x00007f3eb7082250 in dt_film_import () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#7  0x000055e18ba369f4 in main ()

=========

  Id   Target Id                                         Frame 
* 1    Thread 0x7f3eac70ad40 (LWP 21450) "darktable-cli" 0x00007f3eb66473ea in __GI___waitpid (pid=21506, stat_loc=0x0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  2    Thread 0x7f3eac0c0700 (LWP 21461) "gmain"         0x00007f3eb66776d9 in __GI___poll (fds=0x55e18c0a3a90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  3    Thread 0x7f3eab8bf700 (LWP 21462) "gdbus"         0x00007f3eb66776d9 in __GI___poll (fds=0x55e18c0dd770, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7f3e9524e700 (LWP 21504) "lua thread"    0x00007f3eb66776d9 in __GI___poll (fds=0x55e18ccd6970, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  5    Thread 0x7f3e94a4d700 (LWP 21505) "pool"          syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

=========

Thread 5 (Thread 0x7f3e94a4d700 (LWP 21505)):
#0  0x00007f3eb667d219 in syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f3eb6f2675a in g_cond_wait_until () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f3eb6eb2061 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f3eb6f08c12 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f3eb6f08135 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f3eb6550164 in start_thread (arg=<optimized out>) at pthread_create.c:486
        ret = <optimized out>
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139906758530816, -6702681416690687067, 139906766861838, 139906766861839, 139906766861968, 139906758470080, 6667125718457656229, 6667058533091648421}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007f3eb6683def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f3e9524e700 (LWP 21504)):
#0  0x00007f3eb66776d9 in __GI___poll (fds=0x55e18ccd6970, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
        resultvar = 18446744073709551100
        sc_cancel_oldtype = 0
#1  0x00007f3eb6edfe46 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f3eb6ee01d2 in g_main_loop_run () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f3eb719f574 in  () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#4  0x00007f3eb6f08135 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f3eb6550164 in start_thread (arg=<optimized out>) at pthread_create.c:486
        ret = <optimized out>
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139906766923520, -6702681416690687067, 140735212883358, 140735212883359, 140735212883488, 139906766862784, 6667122420459643813, 6667058533091648421}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007f3eb6683def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f3eab8bf700 (LWP 21462)):
#0  0x00007f3eb66776d9 in __GI___poll (fds=0x55e18c0dd770, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
        resultvar = 18446744073709551100
        sc_cancel_oldtype = 0
#1  0x00007f3eb6edfe46 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f3eb6ee01d2 in g_main_loop_run () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f3eb604a7b6 in  () at /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
#4  0x00007f3eb6f08135 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f3eb6550164 in start_thread (arg=<optimized out>) at pthread_create.c:486
        ret = <optimized out>
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139907142776576, -6702681416690687067, 140735212861198, 140735212861199, 140735212861328, 139907142715840, 6667049101609804709, 6667058533091648421}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007f3eb6683def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f3eac0c0700 (LWP 21461)):
#0  0x00007f3eb66776d9 in __GI___poll (fds=0x55e18c0a3a90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
        resultvar = 18446744073709551100
        sc_cancel_oldtype = 0
#1  0x00007f3eb6edfe46 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f3eb6edff6c in g_main_context_iteration () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f3eb6edffb1 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f3eb6f08135 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f3eb6550164 in start_thread (arg=<optimized out>) at pthread_create.c:486
        ret = <optimized out>
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139907151169280, -6702681416690687067, 140735212860798, 140735212860799, 140735212860928, 139907151108544, 6667037007518770085, 6667058533091648421}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#6  0x00007f3eb6683def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f3eac70ad40 (LWP 21450)):
#0  0x00007f3eb66473ea in __GI___waitpid (pid=21506, stat_loc=0x0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
        resultvar = 18446744073709551104
        sc_cancel_oldtype = 0
#1  0x00007f3eb70be150 in  () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#2  0x00007f3eb65aa100 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f3eb70ddd7b in dt_control_progress_create () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#4  0x00007f3eb70d6bd7 in dt_control_job_add_progress () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#5  0x00007f3eb70dd7eb in dt_film_import1_create () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#6  0x00007f3eb7082250 in dt_film_import () at /usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so
#7  0x000055e18ba369f4 in main ()
[Inferior 1 (process 21450) detached]
backtrace written to /tmp/darktable_bt_N5YAWZ.txt
Segmentation fault (core dumped)

@Ruderknecht Ruderknecht mentioned this issue Oct 21, 2019
Closed
@github-actions
Copy link

This issue did not get any activity in the past 30 days and will be closed in 7 days if no update occurs. Please check if the master branch has fixed it since then.

@johnny-bit
Copy link
Member

johnny-bit commented Aug 20, 2020 •
edited

this usecase works perfectly well, here's how you should use dt in your case:

darktable-cli "input_dir" "xmp_file" 'output_dir/$(FILE_NAME).jpg'

and it just Works (tm) :)
you can change ".jpg" to any desired output format handled by darktable.

tested and it works quite well in darktable 3.2.1 as well as 3.0.2 (and current master)

@PetrGlad
Copy link

PetrGlad commented Aug 20, 2020 •
edited

@johnny-bit Thanks! However in any case at least it should not fail with a segmentation fault.
In your example, which OS do you use? In Bash this "output_dir/$(FILE_NAME).jpg" would mean that there should be a command 'FILE_NAME' which gets executed and it's output is captured.

@parafin
Copy link
Member

parafin commented Aug 20, 2020

It should have been darktable-cli "input_dir" "xmp_file" 'output_dir/$(FILE_NAME).jpg'

@johnny-bit
Copy link
Member

@PetrGlad - I've made mistake when copying :) fixed and is identical with @parafin one :)
Additionally it doesn't segfault (that was fixed!)

johnny-bit added a commit to johnny-bit/darktable that referenced this issue Aug 22, 2020
@johnny-bit
allow specifying multiple inputs via --import option
1a79259 
this allow `darktable-cli` to process multiple dirs or selected files
from directory via `--import` option (which can't be used in conjunction
with normal input file param

this essentially fixes darktable-org#2008 for it covers main requirement - multiple inputs
@johnny-bit johnny-bit mentioned this issue Aug 22, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
no-issue-activity
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add couple new features to darktable-cli johnny-bit/darktable
4 participants
@PetrGlad @borconi @parafin @johnny-bit