file.prepend is not acting as expected

[fabbe08]

Description of Issue/Question

file.prepend does not act the way that (I suspect) most people expect it should do. It only adds lines that are unique for the whole file.

Setup

The SLS-file contains:

test:
file.prepend:

• name: /tmp/testfile
• text:
  • AAA
  • BBB

Steps to Reproduce Issue

Before running the formula, the file /tmp/testfile contains this three lines:

D
AA
BBB

After the run, the file contains:

AAA
D
AA
BBB

The line "BBB" is not prepended in the beginning in the file because it exists further down in the file. Only unique lines can be prepended. This make this operation quite useless in many cases, and it is not clear in the documentation that the operation act this way.

It would be better if it always adds all lines, if not every line already exists in the beginning.

Versions Report

Salt Version:
Salt: 2015.8.8.2

Dependency Versions:
Jinja2: 2.7.3
M2Crypto: Not Installed
Mako: Not Installed
PyYAML: 3.11
PyZMQ: 14.4.0
Python: 2.7.9 (default, Mar 1 2015, 12:57:24)
RAET: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5
cffi: 0.8.6
cherrypy: 3.5.0
dateutil: 2.2
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
libgit2: Not Installed
libnacl: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.2
mysql-python: 1.2.3
pycparser: 2.10
pycrypto: 2.6.1
pygit2: Not Installed
python-gnupg: Not Installed
smmap: 0.8.2
timelib: Not Installed

System Versions:
dist: debian 8.3
machine: x86_64
release: 3.16.0-4-amd64
system: debian 8.3

[martin-paulus]

• the states documentation:
  "The text will not be prepended again if it already exists in the file." (https://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html)
• the modules documentation:
  "Prepend text to the beginning of a file" (https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.file.html)

So, the magic happens in the state. The modules' file.py prepend function confirms this.

This looks like it will be fun. My plan is to implement a parameter which toggles between prepending only when unique vs prepending when the lines are not at the top.

My current lead: develop branch, modules/file.py, def replace, line 1872, 1873:

# Search for content, to avoid pre/appending the
# content if it was pre/appended in a previous run.

[fabbe08]

I think it should handle the whole text to be inserted as one entity, not line-by-line. I.e. if I do this with a script, and it found a line "if" somewhere, it should not exclude the line "if" in the beginning.

I was trying to add something to a bash script, and needed a new shebang as a first line (and some new stuff under the shebang), but it refuse to add this because the script already had a shebang (below the stuff I was added, that would be ignored with the new shebang).

The workaround in my case was to make the new shebang a little bit different.

[ssgward]

@fabbe08 - we'll mark this as a feature as it is a change in the current behavior. Looks like @martin-paulus has some new functionality in the works to help with this.

[martin-paulus]

Here is my plan:
The append and prepend functions in modules/file.py are mostly straight forward (except perhaps for the newline check in the append function) and I plan to use them.

For sates/file.py I intend to write two new functions: header and footer. These will scan the (amount 'n' of) user input and then check the first / last 'n' lines of the target file. If these match the user input no modifications to the target will be made. If they do not match, we add the user input above or below the current target file content.

Nice to have (but more involved):
If the evaluated 'n' lines of the target file match the user input partially, add only the missing lines of user input. This should allow modifying files which have previously been targeted, but the user has since decided to expand the user input. Without this check a full new header / footer would be added, since the evaluated 'n' lines do not match the (new) user input. This would duplicate part of the user input in the file.