Semi-transparent windows and rounded corners do not play nice together

[AlexApps99]

Platform

Arch Linux Laptop, linux-lts 5.10.43, picom-git from AUR

GPU, drivers, and screen setup

$ lscpu | grep name
Model name:                      Intel(R) Pentium(R) Silver N5000 CPU @ 1.10GHz
$ pacman -Q | egrep "xf86-video|mesa"
lib32-mesa 21.1.2-1
lib32-vulkan-mesa-layers 21.1.2-1
libva-mesa-driver 21.1.2-1
mesa 21.1.2-1
vulkan-mesa-layers 21.1.2-1
xf86-video-intel 1:2.99.917+916+g31486f40-1
xf86-video-vesa 2.5.0-1
$ glxinfo -B
name of display: :0.0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Intel (0x8086)
    Device: Mesa Intel(R) UHD Graphics 605 (GLK 3) (0x3184)
    Version: 21.1.2
    Accelerated: yes
    Video memory: 3072MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.6
    Max compat profile version: 4.6
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
OpenGL vendor string: Intel
OpenGL renderer string: Mesa Intel(R) UHD Graphics 605 (GLK 3)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 21.1.2
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

OpenGL version string: 4.6 (Compatibility Profile) Mesa 21.1.2
OpenGL shading language version string: 4.60
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile

OpenGL ES profile version string: OpenGL ES 3.2 Mesa 21.1.2
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20

Environment

Xfce 4 (xfwm4)

picom version

$ picom --version
vgit-be24c

Diagnostics

Version: vgit-be24c

Extensions:

• Shape: Yes
• XRandR: Yes
• Present: Present

Misc:

• Use Overlay: No
  (Another compositor is already running)
• Config file used: /home/lxb/.config/picom/picom.conf

Drivers (inaccurate):

Intel

Backend: glx

• Driver vendors:
• GLX: Mesa Project and SGI
• GL: Intel
• GL renderer: Mesa Intel(R) UHD Graphics 605 (GLK 3)
• Accelerated: 1

Configuration:

Configuration file

#################################
#           Corners             #
#################################

# Sets the radius of rounded window corners. When > 0, the compositor will
# round the corners of windows. Does not interact well with
# `transparent-clipping`.
corner-radius = 8

# Exclude conditions for rounded corners.
rounded-corners-exclude = [
  "name = 'Notification'",
  "class_g = 'Conky'",
  "class_g ?= 'Notify-osd'",
  "window_type = 'dock'",
  "window_type = 'desktop'",
  "(_NET_WM_STATE@[0]:a = '_NET_WM_STATE_MAXIMIZED_VERT' || _NET_WM_STATE@[1]:a = '_NET_WM_STATE_MAXIMIZED_VERT' || _NET_WM_STATE@[2]:a = '_NET_WM_STATE_MAXIMIZED_VERT' || _NET_WM_STATE@[3]:a = '_NET_WM_STATE_MAXIMIZED_VERT' || _NET_WM_STATE@[4]:a = '_NET_WM_STATE_MAXIMIZED_VERT') || (_NET_WM_STATE@[0]:a = '_NET_WM_STATE_MAXIMIZED_HORZ' || _NET_WM_STATE@[1]:a = '_NET_WM_STATE_MAXIMIZED_HORZ' || _NET_WM_STATE@[2]:a = '_NET_WM_STATE_MAXIMIZED_HORZ' || _NET_WM_STATE@[3]:a = '_NET_WM_STATE_MAXIMIZED_HORZ' || _NET_WM_STATE@[4]:a = '_NET_WM_STATE_MAXIMIZED_HORZ')",
];


#################################
#             Shadows           #
#################################


# Enabled client-side shadows on windows. Note desktop windows 
# (windows with '_NET_WM_WINDOW_TYPE_DESKTOP') never get shadow, 
# unless explicitly requested using the wintypes option.
#
# shadow = false
shadow = true;

# The blur radius for shadows, in pixels. (defaults to 12)
# shadow-radius = 12

# The opacity of shadows. (0.0 - 1.0, defaults to 0.75)
# shadow-opacity = .75

# The left offset for shadows, in pixels. (defaults to -15)
# shadow-offset-x = -15

# The top offset for shadows, in pixels. (defaults to -15)
# shadow-offset-y = -15

# Specify a list of conditions of windows that should have no shadow.
#
# examples:
#   shadow-exclude = "n:e:Notification";
#
# shadow-exclude = []
shadow-exclude = [
  "name = 'Notification'",
  "class_g = 'Conky'",
  "class_g ?= 'Notify-osd'",
  "window_type = 'dock'",
  "window_type = 'desktop'",
  "_GTK_FRAME_EXTENTS@:c",
];

# Specify a X geometry that describes the region in which shadow should not
# be painted in, such as a dock window region. Use 
#    shadow-exclude-reg = "x10+0+0"
# for example, if the 10 pixels on the bottom of the screen should not have shadows painted on.
#
# shadow-exclude-reg = "" 

#################################
#       General Settings        #
#################################

# Specify the backend to use: `xrender`, `glx`, or `xr_glx_hybrid`.
# `xrender` is the default one.
#
backend = "glx";

# Enable/disable VSync.
# vsync = false
vsync = true

# Try to detect WM windows (a non-override-redirect window with no 
# child that has 'WM_STATE') and mark them as active.
#
# mark-wmwin-focused = false
mark-wmwin-focused = true;

# Mark override-redirect windows that doesn't have a child window with 'WM_STATE' focused.
# mark-ovredir-focused = false
mark-ovredir-focused = true;

# Try to detect windows with rounded corners and don't consider them 
# shaped windows. The accuracy is not very high, unfortunately.
#
# detect-rounded-corners = false
detect-rounded-corners = true;

# Detect '_NET_WM_OPACITY' on client windows, useful for window managers
# not passing '_NET_WM_OPACITY' of client windows to frame windows.
#
# detect-client-opacity = false
detect-client-opacity = true;

# Specify refresh rate of the screen. If not specified or 0, picom will 
# try detecting this with X RandR extension.
#
# refresh-rate = 60
refresh-rate = 0

# Limit picom to repaint at most once every 1 / 'refresh_rate' second to 
# boost performance. This should not be used with 
#   vsync drm/opengl/opengl-oml
# as they essentially does sw-opti's job already, 
# unless you wish to specify a lower refresh rate than the actual value.
#
# sw-opti = 

# Use EWMH '_NET_ACTIVE_WINDOW' to determine currently focused window, 
# rather than listening to 'FocusIn'/'FocusOut' event. Might have more accuracy, 
# provided that the WM supports it.
#
# use-ewmh-active-win = false

# Unredirect all windows if a full-screen opaque window is detected, 
# to maximize performance for full-screen windows. Known to cause flickering 
# when redirecting/unredirecting windows.
#
# unredir-if-possible = false

# Delay before unredirecting the window, in milliseconds. Defaults to 0.
# unredir-if-possible-delay = 0

# Conditions of windows that shouldn't be considered full-screen for unredirecting screen.
# unredir-if-possible-exclude = []

# Use 'WM_TRANSIENT_FOR' to group windows, and consider windows 
# in the same group focused at the same time.
#
# detect-transient = false
detect-transient = true

# Use 'WM_CLIENT_LEADER' to group windows, and consider windows in the same 
# group focused at the same time. 'WM_TRANSIENT_FOR' has higher priority if 
# detect-transient is enabled, too.
#
# detect-client-leader = false
detect-client-leader = true

# Resize damaged region by a specific number of pixels. 
# A positive value enlarges it while a negative one shrinks it. 
# If the value is positive, those additional pixels will not be actually painted 
# to screen, only used in blur calculation, and such. (Due to technical limitations, 
# with use-damage, those pixels will still be incorrectly painted to screen.) 
# Primarily used to fix the line corruption issues of blur, 
# in which case you should use the blur radius value here 
# (e.g. with a 3x3 kernel, you should use `--resize-damage 1`, 
# with a 5x5 one you use `--resize-damage 2`, and so on). 
# May or may not work with *--glx-no-stencil*. Shrinking doesn't function correctly.
#
# resize-damage = 1

# GLX backend: Avoid using stencil buffer, useful if you don't have a stencil buffer. 
# Might cause incorrect opacity when rendering transparent content (but never 
# practically happened) and may not work with blur-background. 
# My tests show a 15% performance boost. Recommended.
#
# glx-no-stencil = false

# GLX backend: Avoid rebinding pixmap on window damage. 
# Probably could improve performance on rapid window content changes, 
# but is known to break things on some drivers (LLVMpipe, xf86-video-intel, etc.).
# Recommended if it works.
#
# glx-no-rebind-pixmap = false

# Disable the use of damage information. 
# This cause the whole screen to be redrawn everytime, instead of the part of the screen
# has actually changed. Potentially degrades the performance, but might fix some artifacts.
# The opposing option is use-damage
#
# no-use-damage = false
use-damage = true

# GLX backend: Use specified GLSL fragment shader for rendering window contents. 
# See `compton-default-fshader-win.glsl` and `compton-fake-transparency-fshader-win.glsl` 
# in the source tree for examples.
#
# glx-fshader-win = ''

# Force all windows to be painted with blending. Useful if you 
# have a glx-fshader-win that could turn opaque pixels transparent.
#
# force-win-blend = false

# Window type settings
# 
# 'WINDOW_TYPE' is one of the 15 window types defined in EWMH standard: 
#     "unknown", "desktop", "dock", "toolbar", "menu", "utility", 
#     "splash", "dialog", "normal", "dropdown_menu", "popup_menu", 
#     "tooltip", "notification", "combo", and "dnd".
# 
# Following per window-type options are available: ::
# 
#   fade, shadow:::
#     Controls window-type-specific shadow and fade settings.
# 
#   opacity:::
#     Controls default opacity of the window type.
# 
#   focus:::
#     Controls whether the window of this type is to be always considered focused. 
#     (By default, all window types except "normal" and "dialog" has this on.)
# 
#   full-shadow:::
#     Controls whether shadow is drawn under the parts of the window that you 
#     normally won't be able to see. Useful when the window has parts of it 
#     transparent, and you want shadows in those areas.
# 
#   redir-ignore:::
#     Controls whether this type of windows should cause screen to become 
#     redirected again after been unredirected. If you have unredir-if-possible
#     set, and doesn't want certain window to cause unnecessary screen redirection, 
#     you can set this to `true`.
#
wintypes:
{
  tooltip = { fade = true; shadow = true; opacity = 0.75; focus = true; full-shadow = false; };
  dock = { shadow = false; }
  dnd = { shadow = false; }
#  popup_menu = { opacity = 0.8; }
#  dropdown_menu = { opacity = 0.8; }
};

Steps of reproduction

1. Open a window with some transparency (For my example I used xfce4-terminal, and xfce4-notifyd, though I also noticed it on xfce4-whiskermenu-plugin)
2. Observe

Expected behavior

No weird triangles on corners of windows

Current Behavior

Weird triangles on corners of windows:
Note that the notification is partway through its fading out animation

[tryone144]

These black corners appear to be from the shadow of the window. Shadows currently don't get cleared pixel perfect (like in the xrender backend) with rounded corners.
You might want to try the wintype option full-shadow = true to disable shadow clipping behind the window, but beware that this darkens the background. Alternatively, wait for rounded corners to be properly implemented in the new backends: #658