Raster Calculator panics (Permission error)

[geotom]

In reference to #197 I experience the same issue with a dockerized Python install. For me the raster_calculator results in the same failure and backtrace. I find it a pity that the raster calculator doesn't work out of the box with a Python install and would hope the Python would get fixed.

By the way, thanks for the great work on these tools!

./whitebox_tools --run="RasterCalculator" --wd="/opt/project/utils/waterbodies/workspace/breached" --statement="raster1" --output='raster2'

thread 'main' panicked at 'failed to execute process: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', whitebox-tools-app/src/tools/mod.rs:1258:26
stack backtrace:
0:     0x5638694f19dc - std::backtrace_rs::backtrace::libunwind::trace::h7f0f727c679b121e
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
1:     0x5638694f19dc - std::backtrace_rs::backtrace::trace_unsynchronized::h27b8207adc787597
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2:     0x5638694f19dc - std::sys_common::backtrace::_print_fmt::h18c4a15687f3f453
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/sys_common/backtrace.rs:67:5
3:     0x5638694f19dc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7077f94cc0b2a619
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/sys_common/backtrace.rs:46:22
4:     0x56386952fdaf - core::fmt::write::hd1fcbdc89ccede50
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/core/src/fmt/mod.rs:1096:17
5:     0x5638694f11e6 - std::io::Write::write_fmt::h3030f95a96ab63e0
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/io/mod.rs:1568:15
6:     0x563869506365 - std::sys_common::backtrace::_print::hce920b6e4ede72f2
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/sys_common/backtrace.rs:49:5
7:     0x563869506365 - std::sys_common::backtrace::print::h63694f3a744810f6
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/sys_common/backtrace.rs:36:9
8:     0x563869506365 - std::panicking::default_hook::{{closure}}::h2f22ad38e825d5e1
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panicking.rs:208:50
9:     0x563869505f33 - std::panicking::default_hook::hc0998dccd7f968a3
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panicking.rs:225:9
10:     0x563869506995 - std::panicking::rust_panic_with_hook::h618d5ded1dbe5ea3
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panicking.rs:591:17
11:     0x5638694f21f7 - std::panicking::begin_panic_handler::{{closure}}::h64f3a351a9df3931
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panicking.rs:497:13
12:     0x5638694f1b3c - std::sys_common::backtrace::__rust_end_short_backtrace::hf7a8c7450479e920
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/sys_common/backtrace.rs:141:18
13:     0x563869506542 - rust_begin_unwind
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panicking.rs:493:5
14:     0x56386952fae1 - core::panicking::panic_fmt::hac8d59bba791921f
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/core/src/panicking.rs:92:14
15:     0x56386952b583 - core::result::unwrap_failed::h386947b3c6c45634
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/core/src/result.rs:1284:5
16:     0x563868c35345 - whitebox_tools::tools::ToolManager::run_tool::h324febd58ab9a2e8
17:     0x563868906abf - whitebox_tools::run::hf0adb91db1a610a3
18:     0x563868903e2a - whitebox_tools::main::h48bb4032c3c51024
19:     0x5638691096f3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h9669996f076d0ae7
20:     0x5638686f4dc9 - std::rt::lang_start::{{closure}}::h4d629acb5d2e5dbf
21:     0x563869506d4a - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hacf21ab16c4aa1f7
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/core/src/ops/function.rs:259:13
22:     0x563869506d4a - std::panicking::try::do_call::h002bd74c4d814788
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panicking.rs:379:40
23:     0x563869506d4a - std::panicking::try::hae0844c0aa736d3d
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panicking.rs:343:19
24:     0x563869506d4a - std::panic::catch_unwind::hec4fe4ae39575e8f
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/panic.rs:431:14
25:     0x563869506d4a - std::rt::lang_start_internal::h024664467cde3297
at /build/rustc-h1hlaa/rustc-1.51.0+dfsg1+llvm/library/std/src/rt.rs:51:25
26:     0x563868907f02 - main
27:     0x7f4f46d5d0b3 - __libc_start_main
28:     0x5638686f4cda - _start
29:                0x0 - <unknown>

[giswqs]

I have not used docker, so I don't know why docker behaves differently than a regular installation using conda. You can check the file permission of the WhiteboxTools binary in the following path and change the permission chmod 755 if needed.

~/.conda/envs/ENV_NAME/lib/python3.x/site-packages/whitebox/whitebox_tools

The whitebox python package should handle the permission during installation. Not sure why it does not work with docker.

https://github.com/giswqs/whitebox-python/blob/master/whitebox/download_wbt.py#L101

            if platform.system() != "Windows":  # grant executable permission
                os.system("chmod 755 " + exe_path)

[geotom]

Hello. Thanks for the quick answer. I have just checked this one again and made sure that all files in "/usr/local/lib/python3.8/dist-packages/whitebox" have the permission 0755. My Docker build does nothing else then installing whitebox from Pypi, then calling a Python script that imports the whitebox tools (and by this triggering the latest binary download). But unfortunately I get the same permission error:

./whitebox_tools --run="RasterCalculator" --wd="/opt/project/utils/waterbodies" --statement="..."
thread 'main' panicked at 'failed to execute process: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', whitebox-tools-app/src/tools/mod.rs:1258:26

I have no problems running other whitebox tools in the same Docker container. Right now I solved my issue with calling the in_place_multiply & subtract tools, but I come to my limits and would need some kind of raster calculation logic.

Looking at the code in whitebox-tools-app/src/tools/mod.rs:1258:26,

  let mut child = Command::new(exe)
      .arg("run")
      .args(&args2)
      .spawn()
      .expect("failed to execute process");

I wonder if this OS Error 13 might be a rust issue and that the arguments to this command are not properly passed?

[giswqs]

Do you mean that all WBT tools work with Python except RasterCalculator? Or is the Python frontend not working at all?

[geotom]

Do you mean that all WBT tools work with Python except RasterCalculator? Or is the Python frontend not working at all?

Yes. Exactly. I use probably ~15 tools I call in a larger Python script (having some other non-related thematical issues still), but none of them throw an error. The raster calculator is the only one panicking.
My docker image is based on "osgeo/gdal:ubuntu-small-latest"

[giswqs]

Then this is probably an issue related to the WBT backend rather than the Python frontend.

[geotom]

I use Docker via WSL2 on Windows. Seems I am not the only one who experienced this issue.

[geotom]

Then this is probably an issue related to the WBT backend rather than the Python frontend.

Anything I could do here?

[giswqs]

Have you tried it directly on Windows without using WSL2?

[geotom]

Sorry for my late replay. I tried directly with a Python 3.8 on Windows 10 and it works as it should. I tried on a vanilla Ubuntu with Python 3.8 in WSL and it failed with a similar error:

thread 'main' panicked at 'Error creating output settings.json file.: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', whitebox-common/src/configs/mod.rs:57:46

[jblindsay]

It looks like your WBT folder is currently in a directory that you do not have write privileges for. The WhiteboxTools executable will attempt to write to the settings.json file contained in the WBT directory when you run a tool.

[geotom]

The command being executed is:

./whitebox_tools --run="RasterCalculator" --wd="/usr/local/lib/python3.8/dist-packages/whitebox" --statement="/opt/project/utils/waterbodies/workspace/dem_breached/dem.RemoveDemDepressions.dem_removed.7e18282af9250c092896cc4ab007b4669928d92a.tif" - 3.0 * "/tmp/tmpgywdh2so/streams_thickened.tif" --output='/opt/project/utils/waterbodies/workspace/dem_burned/dem.BurnDemStreams.burned_dem.5a9d5bed68e0480dc6e2269a23956c80e24d914d.tif'

My observations is that both on a WSL Linux as well as in Docker linux container the default install of Whitebox tools as Python package (Download of the tool) creates a problematic permission. Seems I need to execute my script as root. The WD for instance is the default WD: /usr/local/lib/python3.8/dist-packages/whitebox

[giswqs]

That means the package was incorrectly installed to a system directory rather than a user directory. That's probably the way how docker or WSL was set up. You need to make sure the package is installed to a directory you have write permission to.

[jblindsay]

@giswqs, agreed!

[geotom]

I think I found the issue. I had set my Docker image build script to even set the permissions to 0777 to the folder /usr/local/lib/python3.8/dist-packages/whitebox. The result can be seen here:

#7 260.4 -rwxrwxrwx 1 root staff    175 Nov 22 23:53 __init__.py
#7 260.4 drwxrwsrwx 2 root staff   4096 Nov 22 23:53 __pycache__
#7 260.4 -rwxrwxrwx 1 root staff   3920 Nov 22 23:53 automation.py
#7 260.4 -rwxrwxrwx 1 root staff    416 Nov 22 23:53 cli.py
#7 260.4 -rwxrwxrwx 1 root staff   4950 Nov 22 23:53 download_wbt.py
#7 260.4 -rwxrwxrwx 1 root staff   2469 Nov 22 23:53 example.py
#7 260.4 -rwxrwxrwx 1 root staff  58695 Nov 22 23:53 wb_runner.py
#7 260.4 -rwxrwxrwx 1 root staff    148 Nov 22 23:53 whitebox.py
#7 260.4 -rwxrwxrwx 1 root staff   6146 Nov 22 23:53 whitebox_example.py
#7 260.4 -rwxrwxrwx 1 root staff 426790 Nov 22 23:53 whitebox_tools.py

But I kept on getting the same problem, despite giving all permission possible in the Docker build to the correct folder. When opening a shell to my running container afterwards, I found that I still had folders being created after my Docker image was build. This is how it looks in the running container:

drwxrwsrwx 1 root staff     4096 Nov 22 23:54 .
drwxrwsr-x 1 root staff     4096 Nov 22 23:54 ..
drwxr-sr-x 5 root staff     4096 Nov 22 23:54 WBT
-rw-r--r-- 1 root staff 15383585 Nov 22 23:54 WhiteboxTools_linux_amd64.zip
-rwxrwxrwx 1 root staff      175 Nov 22 23:53 __init__.py
drwxrwsrwx 2 root staff     4096 Nov 22 23:53 __pycache__
-rwxrwxrwx 1 root staff     3920 Nov 22 23:53 automation.py
-rwxrwxrwx 1 root staff      416 Nov 22 23:53 cli.py
-rwxrwxrwx 1 root staff     4950 Nov 22 23:53 download_wbt.py
-rwxrwxrwx 1 root staff     2469 Nov 22 23:53 example.py
drwxr-sr-x 2 root staff     4096 Nov 22 23:54 img
drwxr-sr-x 2 root staff     4096 Nov 22 23:54 plugins
-rw-r--r-- 1 root staff      148 Nov 22 23:54 settings.json
drwxr-sr-x 2 root staff     4096 Nov 22 23:54 testdata
-rwxrwxrwx 1 root staff    58695 Nov 22 23:53 wb_runner.py
-rwxrwxrwx 1 root staff      148 Nov 22 23:53 whitebox.py
-rwxrwxrwx 1 root staff     6146 Nov 22 23:53 whitebox_example.py
-rwxr-xr-x 1 root staff 35129928 Nov 22 23:54 whitebox_tools
-rwxrwxrwx 1 root staff   426790 Nov 22 23:53 whitebox_tools.py

Because the new folders where missing at build time the correct permissions were never set on them and I got the permission error and panic error again.

So far I initialised the Docker image with the following script to make sure the whitebox binary is downloaded already on Docker build time:

RUN echo "import whitebox" > wb.py \
 && python3 wb.py \
 && chmod -R 0777 `find / -type d -name "whitebox"`

I need to find out how to get the additional files and folders being created on in the docker image at build time.
I suspect the whitebox package too also set the 0755 permission too early?

[geotom]

Solution:
I changed the whitebox initialisation in my Dockerfile to:

RUN echo "from whitebox.whitebox_tools import WhiteboxTools" > wb.py \
 && echo "wbt = WhiteboxTools()" >> wb.py \
 && python3 wb.py \
 && chmod -R 0777 `find / -type d -name "whitebox"` 

This downloads all files and when additionally setting the permissions, I have no permission errors/panics in my container later on.

[geotom]

Thanks for all your fast feedback and help. Very much appreciated!