Added support for mixed line ending in file #45498

t0fik · 2018-01-17T14:19:21Z

Adds support for mixed line endings in file.line method.

Test written: No

Not signed

terminalmage

I assume when you say "mixed line ending" you mean windows and unix line endings? If so, this is GTM.

t0fik · 2018-01-19T21:51:02Z

@terminalmage You assume right. What stands for GTM?

sergeizv · 2018-01-19T22:33:20Z

salt/modules/file.py

+    def _split(content):
+        split_content = []
+        for linesep_line in content.split(os.linesep):
+            for content_line in linesep_line.split('\n'):


This code makes a difference only being executed on platforms with os.linesep != '\n'... Why not use str.splitlines() instead?

I'm aware of that.
That specific part of code was tested in file.blockreplace. I do not know how str.splitlines() will behave

sergeizv · 2018-01-19T22:35:45Z

salt/modules/file.py

@@ -1728,11 +1735,11 @@ def line(path, content=None, match=None, mode=None, location=None,
        log.warning('Cannot find text to {0}. File \'{1}\' is empty.'.format(mode, path))
        body = ''
    elif mode == 'delete':
-        body = os.linesep.join([line for line in body.split(os.linesep) if line.find(match) < 0])
+        body = os.linesep.join([line for line in _split(body) if line.find(match) < 0])


Joining with os.linesep and splitting by something else will lead to all the line breaks replaced in the file...

Seems legit to me, to output file with line endings proper to OS.
Converting '\n' to os.linesep is default python behavior for not binary writes.

sergeizv · 2018-01-19T23:02:26Z

On Fri, Jan 19, 2018 at 10:49:01PM +0000, Jerzy Drozdz wrote: t0fik commented on this pull request. > @@ -1728,11 +1735,11 @@ def line(path, content=None, match=None, mode=None, location=None, log.warning('Cannot find text to {0}. File \'{1}\' is empty.'.format(mode, path)) body = '' elif mode == 'delete': - body = os.linesep.join([line for line in body.split(os.linesep) if line.find(match) < 0]) + body = os.linesep.join([line for line in _split(body) if line.find(match) < 0]) Seems legit to me, to output file with line endings proper to OS.

Might be proper for OS, but confusing for software consuming that particular file. Function that modifies a line shouldn't silently change all the file :)

…

Converting '\n' to os.linesep is default python behavior for not binary writes. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #45498 (comment)

t0fik · 2018-01-19T23:44:01Z

@sergeizv What is your proposition, instead converting line endings?
For now that particular function cannot handle line endings, which are not proper for the OS. For example: LF instead CRLF on Windows

sergeizv · 2018-01-20T10:16:47Z

On Fri, Jan 19, 2018 at 11:44:16PM +0000, Jerzy Drozdz wrote: @sergeizv What is your proposition, instead converting line endings?

Maybe detect line break first and then split and join by it instead of os.linesep?

For now that particular function cannot handle line endings, which are not proper for the OS. For example: LF instead CRLF on Windows

The same is true for blockreplace() as well at least

…

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #45498 (comment)

rallytime · 2018-02-02T20:27:43Z

@t0fik and @sergeizv Where did we land on this PR?

t0fik · 2018-02-02T20:53:04Z

I will rewrite line function to detect line endings. I hadn't time to do it.

damon-atkins · 2018-02-03T06:55:24Z

salt/modules/file.py

@@ -1589,6 +1590,13 @@ def _get_line_indent(src, line, indent):

    return ''.join(idt) + line.strip()

+def _get_line_ending(src, line):


This seems to add line ending, rather than get_line_ending consider a different name for the function.
e.g. _add_src_eol(src, line).

@damon-atkins I wanted to be consistent with name scheme of _get_line_indent, which adds indentation to line.
IMHO better names to both functions would be _set_line_eol and _set_line_indent

Good idea, change it

rallytime · 2018-02-06T15:10:06Z

@t0fik There's a lint error: https://jenkins.saltstack.com/job/PR/job/salt-pr-lint-n/19000/violations/

rallytime · 2018-02-06T15:10:35Z

There's also some related test failures: https://jenkins.saltstack.com/job/PR/job/salt-pr-linode-cent7-py3/2006/

damon-atkins · 2018-02-06T15:13:30Z

salt/modules/file.py

@@ -1589,6 +1590,13 @@ def _get_line_indent(src, line, indent):

    return ''.join(idt) + line.strip()

+def _get_line_ending(src, line):


Good idea, change it

…indent and _set_line_eol

terminalmage

I'm not comfortable with something that changes this much to be included in a bugfix release, especially with all the changes to file I/O made for Oxygen. In my opinion this should be done in develop.

terminalmage · 2018-02-08T17:00:06Z

salt/modules/win_file.py

+                               lstat, path_exists_glob, write, pardir, join, HASHES, HASHES_REVMAP,
+                               comment, uncomment, _add_flags, comment_line, _regex_to_static,
+                               _set_line_indent, apply_template_on_contents, dirname, basename,
+                               list_backups_dir, _assert_occurrence, _starts_till, _set_line_eol)


Not sure why you felt the need to indent the entire block here.

It was more readable for me, when I was checking imports. I've reverted indentation to original

terminalmage · 2018-02-08T19:35:43Z

salt/modules/file.py

+    Add line ending
+    '''
+    regex = re.compile('((?<!\r)\n|\r(?!\n)|\r\n)$')
+    line_ending = regex.search(src).group()


What happens on the final line of the file? Not all files end in a newline, and this regex wouldn't match a line without one:

>>> regex = re.compile('((?<!\r)\n|\r(?!\n)|\r\n)$') >>> regex.search('foo').group() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'NoneType' object has no attribute 'group'

I'll add guard statement.

terminalmage · 2018-02-08T19:39:02Z

salt/modules/file.py

    except Exception as ex:
        raise CommandExecutionError("{0}: '{1}'".format(_get_error_message(ex), regex))

-    return src and src.group() or regex
+    return src and src[0] or regex


Why is this being done? I don't see this function being referenced in any of the changed code.

Function is referenced in code which was not changed, line 1734 (orig 1723). I had to change how file is read (to list instead to string) so I had to modify this function

rallytime · 2018-02-28T20:22:20Z

@t0fik I am making the rounds on PRs today :)

Did you get a chance to review the comments above? Those will need to be addressed and this change needs to be moved to develop before we can move forward here.

t0fik · 2018-02-28T21:51:35Z

@rallytime sorry, I was busy. I've reviewed comments and fixed issues.

terminalmage · 2018-03-01T03:40:08Z

This still needs to be rebased/resubmitted on top of develop.

t0fik · 2018-03-01T07:35:44Z

I'll resubmit PR to develop on weekend.

t0fik · 2018-03-02T00:07:45Z

PR resubmited to develop #46291

Added support for mixed line ending in file

3ba0faf

This was referenced Jan 17, 2018

2017.7 line mixed line ends #45500

Closed

Added support for mixed line ending in file (backport of #45498) #45501

Closed

Added support for mixed line endings in file (port of #45498) #45502

Closed

Merge branch '2017.7' into 2017.7_line_mixed_line_ends

c672328

rallytime requested a review from terminalmage January 19, 2018 15:27

terminalmage reviewed Jan 19, 2018

View reviewed changes

sergeizv reviewed Jan 19, 2018

View reviewed changes

t0fik added 5 commits February 2, 2018 22:36

Merge branch '2017.7' into 2017.7_line_mixed_line_ends

7c72149

line function now detects file line endings

3533885

Added _get_line_ending to import in win_file module

25d6769

Fixes after changes

595bdda

Merge branch '2017.7' into 2017.7_line_mixed_line_ends

d2fec06

damon-atkins reviewed Feb 3, 2018

View reviewed changes

Force setting eol

3488139

rallytime requested a review from terminalmage February 6, 2018 15:09

damon-atkins suggested changes Feb 6, 2018

View reviewed changes

t0fik added 4 commits February 6, 2018 18:56

Functions _get_line_indent and _get_line_ending renamed to _set_line_…

e204e5d

…indent and _set_line_eol

pylint problems fixed

6a1f6aa

Fixed "IndexError: list index out of range"

8684710

Merge branch '2017.7' into 2017.7_line_mixed_line_ends

35b74d8

terminalmage reviewed Feb 8, 2018

View reviewed changes

Added guard statement for searching lineseparator

f5fecc4

t0fik added 2 commits February 28, 2018 22:56

Fixed indentation

4d0ec78

Forcing line separator on last line

bbbb8f1

t0fik force-pushed the 2017.7_line_mixed_line_ends branch from 5816a85 to bbbb8f1 Compare February 28, 2018 22:22

t0fik mentioned this pull request Mar 2, 2018

Line mixed line ends #46291

Merged

t0fik closed this Mar 2, 2018

t0fik deleted the 2017.7_line_mixed_line_ends branch March 2, 2018 00:07

t0fik restored the 2017.7_line_mixed_line_ends branch March 4, 2018 02:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for mixed line ending in file #45498

Added support for mixed line ending in file #45498

t0fik commented Jan 17, 2018

terminalmage left a comment

t0fik commented Jan 19, 2018

sergeizv Jan 19, 2018

t0fik Jan 19, 2018 •

edited

sergeizv Jan 19, 2018

t0fik Jan 19, 2018

sergeizv commented Jan 19, 2018 via email

t0fik commented Jan 19, 2018

sergeizv commented Jan 20, 2018 via email

rallytime commented Feb 2, 2018

t0fik commented Feb 2, 2018

damon-atkins Feb 3, 2018

t0fik Feb 4, 2018 •

edited

damon-atkins Feb 6, 2018

rallytime commented Feb 6, 2018

rallytime commented Feb 6, 2018

damon-atkins Feb 6, 2018

terminalmage left a comment

terminalmage Feb 8, 2018

t0fik Feb 28, 2018 •

edited

terminalmage Feb 8, 2018

t0fik Feb 28, 2018

terminalmage Feb 8, 2018

t0fik Feb 28, 2018 •

edited

rallytime commented Feb 28, 2018

t0fik commented Feb 28, 2018 •

edited

terminalmage commented Mar 1, 2018

t0fik commented Mar 1, 2018

t0fik commented Mar 2, 2018

		@@ -1589,6 +1590,13 @@ def _get_line_indent(src, line, indent):

		return ''.join(idt) + line.strip()

		def _get_line_ending(src, line):

Added support for mixed line ending in file #45498

Added support for mixed line ending in file #45498

Conversation

t0fik commented Jan 17, 2018

terminalmage left a comment

Choose a reason for hiding this comment

t0fik commented Jan 19, 2018

Choose a reason for hiding this comment

t0fik Jan 19, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergeizv commented Jan 19, 2018 via email

t0fik commented Jan 19, 2018

sergeizv commented Jan 20, 2018 via email

rallytime commented Feb 2, 2018

t0fik commented Feb 2, 2018

Choose a reason for hiding this comment

t0fik Feb 4, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rallytime commented Feb 6, 2018

rallytime commented Feb 6, 2018

Choose a reason for hiding this comment

terminalmage left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t0fik Feb 28, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t0fik Feb 28, 2018 • edited

Choose a reason for hiding this comment

rallytime commented Feb 28, 2018

t0fik commented Feb 28, 2018 • edited

terminalmage commented Mar 1, 2018

t0fik commented Mar 1, 2018

t0fik commented Mar 2, 2018

t0fik Jan 19, 2018 •

edited

t0fik Feb 4, 2018 •

edited

t0fik Feb 28, 2018 •

edited

t0fik Feb 28, 2018 •

edited

t0fik commented Feb 28, 2018 •

edited