[BUG] `salt-minion 3007.0 (Chlorine)` cannot use `file.append` with a single source (expecting a list object not a string)

[ipaqmaster]

Description
A clear and concise description of what the bug is.

Trying to salt-call state.sls someAppendState causes AttributeError: 'list' object has no attribute 'decode'

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
  KVM and desktop. All are impacted.
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Expected behavior
A clear and concise description of what you expected to happen.

An example state in a file on its own which is falling over:

appender:
  file.append:
    - name: /a
    - source:
      - salt://b

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.0
 
Python Version:
        Python: 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.9.0
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: 0.40.1
          Mako: 1.3.2
       msgpack: 1.0.5
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.2
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.12.0
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: Not Installed
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4
           ZMQ: 4.3.5
 
Salt Package Information:
  Package Type: Not Installed
 
System Versions:
          dist: arch rolling n/a
        locale: utf-8
       machine: x86_64
       release: 6.7.8-arch1-1
        system: Linux
       version: Arch Linux rolling n/a

Additional context
Add any other context about the problem here.

Archlinux 6.7.8-arch1-1

[lkubb]

Has this worked before? From the documentation, source needs to be a single, string-valued source. You can pass a list of files to sources instead.

[ipaqmaster]

I think it clicked for me this morning coming back to this.

I was using - sources: but Salt itself was unhappy with the object being a single string rather than a list object. This checks out with how the source file was working when I appended it with the other file.append state with the other files as a single invocation.

[ipaqmaster]

This also means that when the jinja works out to that single file source - file.append spits No text found to append. Nothing appended instead of appending that file again anyway.

I had to prepend a source:

- salt://blank to prevent this "potential single source situation" and the file also cannot be actually blank otherwise salt says Failed to read rendered template file /var/cache/salt/minion/extrn_files/base/blank (salt://blank) because its empty. Aaaaah.

file.append just needs to be OK if it sees a single source whether its a string or a list. Regardless of whether - source: or - sources: is used.

[whytewolf]

This also means that when the jinja works out to that single file source - file.append spits No text found to append. Nothing appended instead of appending that file again anyway.

I had to prepend a source:

- salt://blank to prevent this "potential single source situation" and the file also cannot be actually blank otherwise salt says Failed to read rendered template file /var/cache/salt/minion/extrn_files/base/blank (salt://blank) because its empty. Aaaaah.

file.append just needs to be OK if it sees a single source whether its a string or a list. Regardless of whether - source: or - sources: is used.

no, source and sources are two separate things and shall always be separate. source does not take a list sources does. sources does work with a single source item when that single source item is passed in as a list of one item. i just tested this

if you don't now if you are going to have multiple sources. use sources and pass a list to it. even if that list only has one item.

if you are always going to use a single source source takes a string.

this is operating as expected.

[ipaqmaster]

I know. Neither work.

[lkubb]

I can't reproduce it either though.

/srv/salt/file1.txt has foo
/srv/salt/append.sls:

create:
  file.touch:
    - names:
      - /tmp/foo_source
      - /tmp/foo_sources

/tmp/foo_source:
  file.append:
    - source: salt://file1.txt

/tmp/foo_sources:
  file.append:
    - sources:
      - salt://file1.txt

show source:
  module.run:
    - file.read:
      - /tmp/foo_source

show sources:
  module.run:
    - file.read:
      - /tmp/foo_sources

Output:

$ salt-call state.apply append
local:
----------
          ID: create
    Function: file.touch
        Name: /tmp/foo_source
      Result: True
     Comment: Created empty file /tmp/foo_source
     Started: 11:20:39.012145
    Duration: 2.138 ms
     Changes:
              ----------
              new:
                  /tmp/foo_source
----------
          ID: create
    Function: file.touch
        Name: /tmp/foo_sources
      Result: True
     Comment: Created empty file /tmp/foo_sources
     Started: 11:20:39.014320
    Duration: 0.272 ms
     Changes:
              ----------
              new:
                  /tmp/foo_sources
----------
          ID: /tmp/foo_source
    Function: file.append
      Result: True
     Comment: Appended 1 lines
     Started: 11:20:39.014619
    Duration: 12.392 ms
     Changes:
              ----------
              diff:
                  ---

                  +++

                  @@ -0,0 +1 @@

                  +foo
----------
          ID: /tmp/foo_sources
    Function: file.append
      Result: True
     Comment: Appended 1 lines
     Started: 11:20:39.027064
    Duration: 7.025 ms
     Changes:
              ----------
              diff:
                  ---

                  +++

                  @@ -0,0 +1 @@

                  +foo
----------
          ID: show source
    Function: module.run
      Result: True
     Comment: file.read: foo
     Started: 11:20:39.034429
    Duration: 0.285 ms
     Changes:
              ----------
              file.read:
                  foo
----------
          ID: show sources
    Function: module.run
      Result: True
     Comment: file.read: foo
     Started: 11:20:39.034743
    Duration: 0.228 ms
     Changes:
              ----------
              file.read:
                  foo

Summary for local
------------
Succeeded: 6 (changed=6)
Failed:    0
------------
Total states run:     6
Total run time:  22.340 ms

[ipaqmaster]

My first jump of a thought is that this may be something specific to the Archlinux packaging of it. But I'll do more testing.

[lkubb]

It might be helpful to see the traceback when running the sls I posted above.

[ocfmatt]

I am receiving the same sort of list decode error when attempting to use file.blockreplace with a list of sources on 3006 or newer on Redhat 8 and 9. I did find #62110 matching my issue directly so this could have some relevancy to this issue.

I can see the file state documentation reads for several sections

A list of sources can also be passed in to provide a default source and a set of fallbacks. The first source in the list that is found to exist will be used and subsequent entries in the list will be ignored.

There is reference to sources being a valid option in append, blockreplace and prepend but this requires all sources to be valid and doesn't have the same fallback logic like the source list has.

if you are always going to use a single source source takes a string.

This statement is only true for append, blockreplace and prepend. I can see the use cases for both source and sources
in their current iteration however, source happily accepts lists on, for example, file.managed making the use and function of the source argument inconsistent.

[lkubb]

Yup, the docs on file.blockreplace are incorrect. It's missing a call to file.source_list for the source parameter before passing it to _unify_sources_and_hashes to match what is documented. It seems the sources parameter is not even documented, but operates as described (requires and concatenates all sources in the list).

I receive the traceback this issue is about on both file.blockreplace and file.append only when passing a list of strings to source (singular, like in the referenced file.blockreplace issue), but OP has stated specifically that he's not doing that, so both issues should (tm) not be related.

Agree that all file states should have the same parameter handling (where sensible):

• All source parameters should be able to handle strings or lists, where lists are used to find a single existing source.
• All sources parameters should operate with and require all specified sources. (I think that's the case already)