[BUG] file.managed function leads to "Unable to manage file: '__file_client__'"

[lallish]

Description
I upgraded from 3006.1 to 3006.5 and after upgrade all file.managed does not work anymore. When I use the function file.managed I get this error for example:

ID: docker-service-file
Function: file.managed
Name: /etc/systemd/system/docker.service
Result: False
Comment: Unable to manage file: '__file_client__'
Started: 07:25:19.738565
Duration: 4.006 ms

Setup

roots.conf

file_roots:
  base:
    - /srv/salt/dev
    - /srv/salt/base
    - /srv/files/dev
    - /srv/files/base

pillar_roots:
  base:
    - /srv/pillar/dev
    - /srv/pillar/base

env_order: ['base']
default_top: base

Please be as specific as possible and give set-up details.

• AWS EC2-machine
• Salt as systemd service
• Amazon Linux 2
• onedir packaging
• used yum install to install salt

Steps to Reproduce the behavior

1. Run 3006.1 on salt-master and salt-minion.
2. Install 3006.5 on salt-master and salt-minion. Restart systemd services on both.
3. Run a state where with file.managed copy a file from salt-master as source, e.g.

download_java:
  file.managed:
    - name: /tmp/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz
    - source: salt://files/java_downloads/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz

4. Run from salt-master targeting the salt-minion, such as sudo salt '<salt-minion>' state.apply <state>
5. Error shows up:

ID: download_java
Function: file.managed
Name: /tmp/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz
Result: False
Comment: Unable to manage file: '__file_client__'
Started: 07:48:28.500227
Duration: 10.971 ms
Changes:

Expected behavior
The file should be copied to the destination.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt-master:

Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 5.0.2
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.6.4
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.12.1
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: amzn 2
        locale: utf-8
       machine: x86_64
       release: 4.14.314-238.539.amzn2.x86_64
        system: Linux
       version: Amazon Linux 2

Salt-minion:

Salt Version:
           Salt: 3006.5

 Python Version:
         Python: 3.10.13 (main, Nov 15 2023, 04:37:53) [GCC 11.2.0]

 Dependency Versions:
           cffi: 1.14.6
       cherrypy: unknown
       dateutil: 2.8.1
      docker-py: 1.10.6
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 3.1.2
        libgit2: Not Installed
   looseversion: 1.0.2
       M2Crypto: Not Installed
           Mako: Not Installed
        msgpack: 1.0.2
   msgpack-pure: Not Installed
   mysql-python: Not Installed
      packaging: 22.0
      pycparser: 2.21
       pycrypto: Not Installed
   pycryptodome: 3.9.8
         pygit2: Not Installed
   python-gnupg: 0.4.8
         PyYAML: 6.0.1
          PyZMQ: 23.2.0
         relenv: 0.14.2
          smmap: Not Installed
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.3.4

 System Versions:
           dist: amzn 2
         locale: utf-8
        machine: aarch64
        release: 4.14.328-248.540.amzn2.aarch64
         system: Linux
        version: Amazon Linux 2

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[lallish]

Downgraded to 3006.3 and the problem isn't found anymore. So either I missed some breaking change in the changelog or the culprit is in 3006.4 or 3006.5.

[max-arnold]

Have you tried a clean install of 3006.5 or 3006.6?

[lallish]

Have you tried a clean install of 3006.5 or 3006.6?

Nope, only upgrade from 3006.1. It was installed through a normal yum install so yum should clean up the previous version prior to installing.

[dwoz]

I was not able to reproduce this with 3006.5

salt -c etc/master1 minion0001 state.apply file
[INFO    ] User dan Published command state.apply with jid 20240208215202186955
[INFO    ] User dan Executing command state.apply with jid 20240208215202186955
[INFO    ] Starting a new job 20240208215202186955 with PID 15861
[INFO    ] Executing command 'echo' in directory '/home/dan'
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/tmp/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz] at time 14:52:02.424143
[INFO    ] Executing state file.managed for [/tmp/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz]
[INFO    ] File changed:
New file
[INFO    ] Completed state [/tmp/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz] at time 14:52:02.435640 (duration_in_ms=11.497)
[INFO    ] Returning information for job: 20240208215202186955
[INFO    ] Got return from minion0001 for job 20240208215202186955
minion0001:
----------
          ID: download_java
    Function: file.managed
        Name: /tmp/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz
      Result: True
     Comment: File /tmp/amazon-corretto-21.0.1.12.1-linux-x64.tar.gz updated
     Started: 14:52:02.424143
    Duration: 11.497 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0644

Summary for minion0001
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  11.497 ms

[dwoz]

@lallish Can you please include the debug logs from a minion while running your state?