Use of xdotool type with shell type var breaks in 2.1.1-alpha

[jtmackoy]

Describe the bug
Please reference #742 for the workaround implemented there - it elaborates on the workaround where I needed to invoke xdotool type to address a character case problem (lack of SHIFT+<character> when expanding in certain applications.

After upgrading to 2.1.1-alpha, and going through the migration process that moved my %CONFIG%/default.yml to %CONFIG%/match/base.yml, none of my triggers would work.

To Reproduce
Steps to reproduce the behavior:

1. Stop, unregister, then uninstall espanso 0.7.3
2. Install, register, start espanso 2.1.1-alpha
3. Attempt a trigger (with the workaround referenced above as part of base.yml)
4. Nothing is expanded/replaced

Expected behavior
Step 3 above would result in a successful expansion

Screenshots
None - see command examples referenced throughout.

Logs
Note: I've replaced some paths with %CONFIG% below, others with ~ that reference my home directory.

21:19:00 [daemon(31495)] [WARN] ------- detected some errors in the configuration: -------
21:19:00 [daemon(31495)] [WARN] >>> %CONFIG%/match/base.yml
21:19:00 [daemon(31495)] [ERROR] unable to load match group "%CONFIG%/match/base.yml"

Caused by:
    0: failed to parse YAML match group
    1: while scanning a quoted scalar, found unexpected end of stream at line 55 column 54
    2: while scanning a quoted scalar, found unexpected end of stream at line 55 column 54
21:19:00 [daemon(31495)] [WARN] -----------------------------------------------------------
21:19:00 [daemon(31495)] [INFO] espanso version: 2.1.1-alpha
21:19:00 [daemon(31495)] [INFO] spawning the worker process...
21:19:00 [daemon(31495)] [INFO] binded to IPC unix socket: ~/.local/share/espanso/espansodaemonv2.sock

In my %CONFIG%/match/base.yml file, line 55 is as follows:

  - trigger: ":my_pw"
    replace: "{{output}}"
    vars:
      - name: output
        type: shell
        params:
-->       cmd: "xdotool type 'ThIsIsCoMpLiCaTeD!,>@;"'"
          trim: false

Setup information

• OS:

cat /etc/os-release
NAME="Pop!_OS"
VERSION="21.04"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 21.04"
VERSION_ID="21.04"
HOME_URL="https://pop.system76.com"
SUPPORT_URL="https://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute
LOGO=distributor-logo-pop-os

• Version: which version of Espanso are you running? (you can find out by running espanso --version inside a terminal)

espanso --version
espanso 2.1.1-alpha

** Additional info**

I've tried a number of different approaches to solving this problem under v2.1.1-alpha. Ultimately, I suspect that the issue is xdotool type inserts some sort of character at the end of its output that the new version of espanso doesn't like:

❯ xdotool type 'ThIsIsCoMpLiCaTeD!,>@;"'
ThIsIsCoMpLiCaTeD!,>@;"%   #the result of the command, note the "%" at the end
❯ ThIsIsCoMpLiCaTeD!,>@;"   #the text xdotool types out

I believe it's a new line or carriage return, but attempting to remove any references to those characters causes odd behavior when executing a trigger - although there are no more failed to parse YAML match group errors using the modified cmd.

Via shell (zsh) running the following by using tr -d to remove those types of characters still appears to show that % character in the output line, before the text is typed as I would expect:

❯ xdotool type -- "$(echo 'ThIsIsCoMpLiCaTeD!,>@;"' | tr -d \r | tr -d \n)"
ThIsIsCoMpLiCaTeD!,>@;"%
❯ ThIsIsCoMpLiCaTeD!,>@;"

And when I add that line to the cmd for my :my_pw trigger, here is the expansion behavior I see:
typing ❯ :my_pw
results in the following expansion ❯ :my_pwThIsIsCoMpLiCaTe

As a result of this break, I've had to downgrade back to espanso 0.7.3. Hopefully the filter that can be produced on #742 will help to sort this out.

[federico-terzi]

Hey @jtmackoy,

I'm sorry you experienced this problem!

21:19:00 [daemon(31495)] [WARN] ------- detected some errors in the configuration: -------
21:19:00 [daemon(31495)] [WARN] >>> %CONFIG%/match/base.yml
21:19:00 [daemon(31495)] [ERROR] unable to load match group "%CONFIG%/match/base.yml"

Caused by:
    0: failed to parse YAML match group
    1: while scanning a quoted scalar, found unexpected end of stream at line 55 column 54
    2: while scanning a quoted scalar, found unexpected end of stream at line 55 column 54
21:19:00 [daemon(31495)] [WARN] -----------------------------------------------------------
21:19:00 [daemon(31495)] [INFO] espanso version: 2.1.1-alpha
21:19:00 [daemon(31495)] [INFO] spawning the worker process...
21:19:00 [daemon(31495)] [INFO] binded to IPC unix socket: ~/.local/share/espanso/espansodaemonv2.sock

This is interesting, it seems the migration tool produced some invalid YAML... Keep in mind that if the YAML is invalid, none of those matches are loaded, so that would explain why Espanso is not expanding any of your matches.

In particular, this line contains a doublequote that has not been escaped:

cmd: "xdotool type 'ThIsIsCoMpLiCaTeD!,>@;"'"

which should be (notice the \ before the ")

cmd: "xdotool type 'ThIsIsCoMpLiCaTeD!,>@;\"'"

That said, this should not happen. I'll investigate, thank you for the help :)

Cheers :)

[federico-terzi]

@jtmackoy I've investigated a bit, but unfortunately I wasn't able to trigger this bug in the migration process. I understand this is confidential information, but it would be a great help to find a match that causes this migration problem (before the migration, so that we can isolate a specific migration bug, if present)

Do you have any idea?

[jtmackoy]

@federico-terzi - sorry for being a stranger! This issue just cropped up again after I was in a position to reinstall espanso (after an OS upgrade to Pop!-OS 22.04) and found that the current 2.1.5-beta is the recommended version to use.

As before, after the upgrade, I ran into issues using a shell type with xdotool type '...' for the cmd argument. As before, the substitution seemed to leave the trigger string in place, inserted the expanded text after the trigger string, then trimmed the same number of characters from the tail of the expanded text as are used by the trigger string.

This problem still seemed to be unique to the use of xdotool type as any other commands I replaced in the cmd argument were able to work just fine (trigger string was replaced with expanded text).

For context, my rationale for needing to use xdotool type is discussed under #742.

Ultimately I found a different method to accomplish my goal:

this:

  - trigger: ":my_pw"
    replace: "{{output}}"
    vars:
      - name: output
        type: shell
        params:
          cmd: "xdotool type '$(secret-tool lookup credential password)'"
          trim: false

(the value of that secret-tool lookup was equivalent to ThIsIsCoMpLiCaTeD!,>@;")

was replaced with this:

  - trigger: ":my_pw"
    replace: "{{output}}"
    vars:
      - name: output
        type: shell
        params:
          cmd: secret-tool lookup credential password

and

I updated $CONFIG/config/default.yml with this:

{
  backend: Clipboard
}

So, now, I'm able to execute :my_pw anywhere - including inside my QEMU/KVM VMs and the pasted value is always correct (not missing characters and case is as I expect).

Not sure if this constitutes a resolution to either this issue or #742, but thought I'd share my shift in approach to this problem.

Also - in case no one's told you today: Thanks for the great tool!

[smeech]

I'll close this as I think it's resolved for now. I don't know if the newly documented x11_use_xdotool_backend option is of help?