x11 + GNOME/Wayland support #102

mokurin000 · 2023-01-08T15:52:09Z

implement is_rmb_down for linux/x11

PedroHLC · 2023-01-09T11:44:01Z

I can confirm that it builds with newer deps libxcb and libxdo (from xdotools).

There is a silent dependency on xwininfo otherwise it fails with:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: Empty }', src/main.rs:210:49

Once you start it, it changes your cursor over X11 windows to a "cross" cursor, but when I click in the GI's window, it fails with:

[2023-01-09T11:42:32Z INFO  yas] left = 0, top = 0, width = 3840, height = 2160
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: failed to decode Protobuf message: invalid wire type value: 6', src/inference/inference.rs:33:52
stack backtrace:
   0:     0x55bb9a5b8e73 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he3010f3ff51b55ac
   1:     0x55bb9a5f53dc - core::fmt::write::hdfaf46c08a67cd92
   2:     0x55bb9a5d2ad5 - std::io::Write::write_fmt::h0c8cb8452451d559
   3:     0x55bb9a5d95d4 - std::panicking::default_hook::{{closure}}::h5f89d231703325b8
   4:     0x55bb9a5d9214 - std::panicking::default_hook::h0daa4414d4591360
   5:     0x55bb9a5d9bb3 - std::panicking::rust_panic_with_hook::hc7eecadc23f906e1
   6:     0x55bb9a5b91d7 - std::panicking::begin_panic_handler::{{closure}}::h425c6c4810da3633
   7:     0x55bb9a5b8fb4 - std::sys_common::backtrace::__rust_end_short_backtrace::h75a4c4b09ed50e09
   8:     0x55bb9a5d98a2 - rust_begin_unwind
   9:     0x55bb988baab3 - core::panicking::panic_fmt::hed5cc02befa9fbd6
  10:     0x55bb988babe3 - core::result::unwrap_failed::he50d59caf960046c
  11:     0x55bb9892859c - core::result::Result<T,E>::unwrap::h39a02265ef19c265
  12:     0x55bb988f1955 - yas::inference::inference::CRNNModel::new::hb2bc588d872a6249
  13:     0x55bb988ea7c8 - yas::scanner::yas_scanner::YasScanner::new::hec4916971100483b
  14:     0x55bb988bed94 - yas::main::h5b2d9428909c513b
  15:     0x55bb988baecb - core::ops::function::FnOnce::call_once::hab5d454436d5a6b5
  16:     0x55bb988bb99e - std::sys_common::backtrace::__rust_begin_short_backtrace::h04929074ddc4d974
  17:     0x55bb988c0841 - std::rt::lang_start::{{closure}}::hce1e3522327f47a0
  18:     0x55bb9a5d1f08 - std::rt::lang_start_internal::h0b59928b60b7c3a2
  19:     0x55bb988c081a - std::rt::lang_start::h357c7da2a3399e8f
  20:     0x55bb988bf881 - main
  21:     0x7f4c228fe24e - __libc_start_call_main
  22:     0x7f4c228fe309 - __libc_start_main_alias_1
  23:     0x55bb988badd5 - _start
  24:                0x0 - <unknown>

Could it have disliked the fact that I'm on XWayland?

mokurin000 · 2023-01-09T14:11:14Z

did you run git lfs pull before building

        let mut bytes = include_bytes!("../../models/model_acc100-epoch45.onnx");
        // let mut bytes = include_bytes!("../../models/model_training.onnx");

        let model = tract_onnx::onnx()
            .model_for_read(&mut bytes.as_bytes()).unwrap()
            .with_input_fact(0, InferenceFact::dt_shape(f32::datum_type(), tvec!(1, 1, 32, 384))).unwrap()
            .into_optimized().unwrap()
            .into_runnable().unwrap();

the model file should be fetched with Git LFS
btw, screenshots-rs has bad performance on Wayland
idk if you can screenshot as using X11 with Xwayland, but when I tried unset WAYLAND_DISPLAY it fails.

PedroHLC · 2023-01-09T14:58:26Z

did you run git lfs pull before building

Sorry, that was indeed the issue.

idk if you can screenshot as using X11 with Xwayland
but when I tried unset WAYLAND_DISPLAY it fails.

Was I utilising the Wayland mode of it without realizing it? (with WAYLAND_DISPLAY set)
Anyway, it works with/without WAYLAND_DISPLAY for me. But I always have zero results:

╰─λ ./result/bin/yas --verbose --output-dir /tmp/yas/
[2023-01-09T14:55:04Z INFO  yas] left = 0, top = 0, width = 3840, height = 2160
[2023-01-09T14:55:05Z INFO  tract_linalg::x86_64_fma] mmm_f32, sigmoid_f32, tanh_f32: x86_64/fma activated
[2023-01-09T14:55:05Z INFO  tract_linalg::x86_64_fma] mmm_i8_i8 and mmm_i8_i32: x86_64/avx2 activated
^C⏎                                                                                                                                                                                                                  ╭─pedrohlc at desktop in ~/yas (linux-x11-support …2)
╰─λ la /tmp/yas/
total 0
╭─pedrohlc at desktop in ~/yas (linux-x11-support …2)
╰─λ env | grep DISPLAY
DISPLAY=:0

╰─λ ./result/bin/yas --verbose --output-dir /tmp/yas/
[2023-01-09T14:57:01Z INFO  yas] left = 0, top = 0, width = 3840, height = 2160
[2023-01-09T14:57:02Z INFO  tract_linalg::x86_64_fma] mmm_f32, sigmoid_f32, tanh_f32: x86_64/fma activated
[2023-01-09T14:57:02Z INFO  tract_linalg::x86_64_fma] mmm_i8_i8 and mmm_i8_i32: x86_64/avx2 activated
^C⏎                                                                                                                                                                                                                  ╭─pedrohlc at desktop in ~/yas (linux-x11-support …2)
╰─λ la /tmp/yas/
total 0
╭─pedrohlc at desktop in ~/yas (linux-x11-support …2)
╰─λ env | grep DISPLAY
DISPLAY=:0
WAYLAND_DISPLAY=wayland-1

mokurin000 · 2023-01-09T15:36:51Z

edroHLC

may you pls record repreduce steps for me?..
I guess your game's UI language was not Chinese, In that case yas do not work at all, as mentioned in #92

and sorry for my misinformation, screenshots-rs detects wayland session by both XDG_SESSION_TYPEand WAYLAND_DISPLAY.

~/Source/yas linux-x11-support > unset WAYLAND_DISPLAY
~/Source/yas linux-x11-support > export XDG_SESSION_TYPE=x11                                                                                                                                       23:48:49
~/Source/yas linux-x11-support > ./target/release/yas -f good --number 5                                                                                                                           23:49:25
[2023-01-09T15:49:45Z INFO  yas] left = 280, top = 90, width = 1600, height = 900
[2023-01-09T15:49:45Z INFO  tract_linalg::x86_64_fma] mmm_f32, sigmoid_f32, tanh_f32: x86_64/fma activated
[2023-01-09T15:49:45Z INFO  tract_linalg::x86_64_fma] mmm_i8_i8 and mmm_i8_i32: x86_64/avx2 activated
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "capture failed"', src/capture/mod.rs:63:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[1]    19375 IOT instruction (core dumped)  ./target/release/yas -f good --number 5

PedroHLC · 2023-01-09T15:58:12Z

I guess your game's UI language was not Chinese

It was not, but same result in both Chinese.

and sorry for my misinformation, screenshots-rs detects wayland session by both XDG_SESSION_TYPEand WAYLAND_DISPLAY.

Ok, now I get the same crash as you.

UPDATE: Also, the issue with XWayland seems to be time to capture+decode as you said.

…-x11-support

mokurin000 · 2023-01-10T12:57:07Z

I guess your game's UI language was not Chinese

It was not, but same result in both Chinese.

and sorry for my misinformation, screenshots-rs detects wayland session by both XDG_SESSION_TYPEand WAYLAND_DISPLAY.

Ok, now I get the same crash as you.

UPDATE: Also, the issue with XWayland seems to be time to capture+decode as you said.

btw, it will call org.freedesktop.portal.ScreenShot
and decode a full 1980*1080 image to pick one pixel

Could you help me to implement a better pick_color for Wayland with PickColor method

remove debug config in Cargo.toml update Cargo.lock

…-x11-support

PedroHLC · 2023-01-27T13:50:45Z

With or without WAYLAND_DISPLAY, current commit fails:

╰─λ RUST_BACKTRACE=full ./result/bin/yas --verbose --output-dir /tmp/
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: Empty }', src/main.rs:210:49
stack backtrace:
   0:     0x559165737682 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h0b60020a4713e5fb
   1:     0x559165772d0c - core::fmt::write::hd2bbf674326f971d
   2:     0x55916574b635 - std::io::Write::write_fmt::h9e09deb1651b82d4
   3:     0x559165737441 - std::sys_common::backtrace::print::hb0f6f2fe5c4496c1
   4:     0x559165758f1f - std::panicking::default_hook::{{closure}}::hf3d6d483953ab294
   5:     0x559165758ba4 - std::panicking::default_hook::h7229a04a9f746202
   6:     0x559165759559 - std::panicking::rust_panic_with_hook::h7c53bfc9ab300aca
   7:     0x5591657379d7 - std::panicking::begin_panic_handler::{{closure}}::hc2a71b5a463ec201
   8:     0x5591657377b4 - std::sys_common::backtrace::__rust_end_short_backtrace::he21bec3aa77e6ef6
   9:     0x5591657590d2 - rust_begin_unwind
  10:     0x559163a2a263 - core::panicking::panic_fmt::h136a2a3f69fc0251
  11:     0x559163a2a623 - core::result::unwrap_failed::h656bdbcac2f79cc7
  12:     0x559163a2f4b0 - core::result::Result<T,E>::unwrap::h06bff2424fe9a9de
  13:     0x559163a2cf9a - yas::main::h09d2d47b704854de
  14:     0x559163a2b3fb - core::ops::function::FnOnce::call_once::h2761a5f2a2c695da
  15:     0x559163a2b22e - std::sys_common::backtrace::__rust_begin_short_backtrace::h7f004f0fc531b600
  16:     0x559163a2b3b1 - std::rt::lang_start::{{closure}}::h51cb07d8dc8bc397
  17:     0x55916574a484 - std::rt::lang_start_internal::h398023bd7970a80b
  18:     0x559163a2b38a - std::rt::lang_start::h42181c3b39454930
  19:     0x559163a2e70e - main
  20:     0x7f11f36c124e - __libc_start_call_main
  21:     0x7f11f36c1309 - __libc_start_main_alias_1
  22:     0x559163a2a6f5 - _start
  23:                0x0 - <unknown>

(Sorry, I was traveling last weeks)

mokurin000 · 2023-01-30T21:27:19Z

With or without WAYLAND_DISPLAY, current commit fails:

╰─λ RUST_BACKTRACE=full ./result/bin/yas --verbose --output-dir /tmp/
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: Empty }', src/main.rs:210:49
stack backtrace:
   0:     0x559165737682 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h0b60020a4713e5fb
   1:     0x559165772d0c - core::fmt::write::hd2bbf674326f971d
   2:     0x55916574b635 - std::io::Write::write_fmt::h9e09deb1651b82d4
   3:     0x559165737441 - std::sys_common::backtrace::print::hb0f6f2fe5c4496c1
   4:     0x559165758f1f - std::panicking::default_hook::{{closure}}::hf3d6d483953ab294
   5:     0x559165758ba4 - std::panicking::default_hook::h7229a04a9f746202
   6:     0x559165759559 - std::panicking::rust_panic_with_hook::h7c53bfc9ab300aca
   7:     0x5591657379d7 - std::panicking::begin_panic_handler::{{closure}}::hc2a71b5a463ec201
   8:     0x5591657377b4 - std::sys_common::backtrace::__rust_end_short_backtrace::he21bec3aa77e6ef6
   9:     0x5591657590d2 - rust_begin_unwind
  10:     0x559163a2a263 - core::panicking::panic_fmt::h136a2a3f69fc0251
  11:     0x559163a2a623 - core::result::unwrap_failed::h656bdbcac2f79cc7
  12:     0x559163a2f4b0 - core::result::Result<T,E>::unwrap::h06bff2424fe9a9de
  13:     0x559163a2cf9a - yas::main::h09d2d47b704854de
  14:     0x559163a2b3fb - core::ops::function::FnOnce::call_once::h2761a5f2a2c695da
  15:     0x559163a2b22e - std::sys_common::backtrace::__rust_begin_short_backtrace::h7f004f0fc531b600
  16:     0x559163a2b3b1 - std::rt::lang_start::{{closure}}::h51cb07d8dc8bc397
  17:     0x55916574a484 - std::rt::lang_start_internal::h398023bd7970a80b
  18:     0x559163a2b38a - std::rt::lang_start::h42181c3b39454930
  19:     0x559163a2e70e - main
  20:     0x7f11f36c124e - __libc_start_call_main
  21:     0x7f11f36c1309 - __libc_start_main_alias_1
  22:     0x559163a2a6f5 - _start
  23:                0x0 - <unknown>

(Sorry, I was traveling last weeks)

well.. this error means you did not correctly click GI for xwininfo
also, iirc, I haven't fix gnome/wayland screenshot yet.. sry

wormtql

为啥要把version搞成env的形式，不从cargo.toml读取

wormtql · 2023-02-12T05:37:02Z

此外为啥要用你fork的screenshot而不是原版

mokurin000 · 2023-02-12T15:49:36Z

为啥要把version搞成env的形式，不从cargo.toml读取

https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-crates
编译器提供的编译时环境变量

这样编译时可以直接拿到

mokurin000 · 2023-02-12T15:52:57Z

此外为啥要用你fork的screenshot而不是原版

原版的screenshots-rs在每个平台都把拿到的截图编码为png rgba8再给…… 然后你的screenshot这边我看了下是要一份bgra raw image？

mokurin000 added 14 commits January 8, 2023 20:27

get version from CARGO_PKG_VERSION at compile time

fc11847

use screenshots-rs to capture screen

b54148b

add attribute for win-only functions

5affb3e

Merge branch 'use-cargo-pkg-ver' into linux-x11-support

c79ad9e

leave it as TODO

72c5f7a

initial support without return to break

27d0111

remove revert in crop

4480fb1

remove reverse in to_gray

0adb220

prevent unnessacary PNG decode/encode

5f3d7ad

fix is_bgra

236612d

removed failed temp capture

32ff8cd

fix windows build

10f9b2e

todo: fix mouse scroll

629fcc7

fix: mouse scroll on x11

94d6231

mokurin000 marked this pull request as ready for review January 9, 2023 07:30

mokurin000 and others added 3 commits January 9, 2023 15:41

添加linux/x11下使用说明

f1f3ff4

Merge commit 'f1f3ff4' into linux-x11-support

a03169d

Update README.md

4d8b9e0

mokurin000 added 2 commits January 9, 2023 22:18

add build dep for linux

376461c

Update README.md

038cc50

mokurin000 added 3 commits January 10, 2023 03:57

rename get_color to get_flag_color

f4aca95

Merge branch 'linux-x11-support' of github.com:poly000/yas into linux…

22c7be5

…-x11-support

bump deps

f0119ec

fix screenshots on GNOME/Wayland

a716391

mokurin000 and others added 4 commits January 11, 2023 14:59

Update README.md

cc7f5c2

remove debug config in Cargo.toml

a0049b1

Update README.md

ee2864f

remove debug config in Cargo.toml update Cargo.lock

Merge branch 'linux-x11-support' of github.com:poly000/yas into linux…

afcfa6a

…-x11-support

mokurin000 changed the title ~~Linux x11 support~~ x11 + GNOME/Wayland support Jan 13, 2023

wormtql reviewed Feb 12, 2023

View reviewed changes

wormtql merged commit 292e852 into wormtql:main Feb 13, 2023

Omico mentioned this pull request Jun 17, 2023

yas_v0.1.11可以使用但yas_v0.1.13无法使用 #134

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x11 + GNOME/Wayland support #102

x11 + GNOME/Wayland support #102

mokurin000 commented Jan 8, 2023 •

edited

PedroHLC commented Jan 9, 2023

mokurin000 commented Jan 9, 2023 •

edited

PedroHLC commented Jan 9, 2023 •

edited

mokurin000 commented Jan 9, 2023 •

edited

PedroHLC commented Jan 9, 2023 •

edited

mokurin000 commented Jan 10, 2023 •

edited

PedroHLC commented Jan 27, 2023

mokurin000 commented Jan 30, 2023

wormtql left a comment

wormtql commented Feb 12, 2023

mokurin000 commented Feb 12, 2023 •

edited

mokurin000 commented Feb 12, 2023 •

edited

x11 + GNOME/Wayland support #102

x11 + GNOME/Wayland support #102

Conversation

mokurin000 commented Jan 8, 2023 • edited

PedroHLC commented Jan 9, 2023

mokurin000 commented Jan 9, 2023 • edited

PedroHLC commented Jan 9, 2023 • edited

mokurin000 commented Jan 9, 2023 • edited

PedroHLC commented Jan 9, 2023 • edited

mokurin000 commented Jan 10, 2023 • edited

PedroHLC commented Jan 27, 2023

mokurin000 commented Jan 30, 2023

wormtql left a comment

Choose a reason for hiding this comment

wormtql commented Feb 12, 2023

mokurin000 commented Feb 12, 2023 • edited

mokurin000 commented Feb 12, 2023 • edited

mokurin000 commented Jan 8, 2023 •

edited

mokurin000 commented Jan 9, 2023 •

edited

PedroHLC commented Jan 9, 2023 •

edited

mokurin000 commented Jan 9, 2023 •

edited

PedroHLC commented Jan 9, 2023 •

edited

mokurin000 commented Jan 10, 2023 •

edited

mokurin000 commented Feb 12, 2023 •

edited

mokurin000 commented Feb 12, 2023 •

edited