Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CR duplication in ASCII mode download #492

Merged
merged 4 commits into from Mar 27, 2019
Merged

Fix CR duplication in ASCII mode download #492

merged 4 commits into from Mar 27, 2019

Conversation

andrewshulgin
Copy link
Contributor

@andrewshulgin andrewshulgin commented Mar 26, 2019

If a file has CRLF already, LF has been still replaced by CRLF, which resulted in CRCRLF line endings.

@coveralls
Copy link

coveralls commented Mar 26, 2019

Coverage Status

Coverage remained the same at ?% when pulling 52db1bf on andrew-shulgin:ascii_crlf_fix into 9cf85c1 on giampaolo:master.

@lurch
Copy link
Contributor

lurch commented Mar 27, 2019

How does this compare with what other FTP servers do in ASCII mode?

pos = chunk.find(b'\n', pos)
if pos == -1:
break
if chunk[pos - 1] != 13:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's 13?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is ord('\r'). I guess you'll suggest to create a constant like CR = ord('\r'). Am I correct?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. I would define it at the top of this method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way, ord('\r') will be executed each time the method is invoked.

I defined CR_BYTE at the top of handlers.py, which I believe is better. Is this ok?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. Yeah leave it there. =)

@andrewshulgin
Copy link
Contributor Author

How does this compare with what other FTP servers do in ASCII mode?

PureFTPd has the same issue.
https://support.microfocus.com/kb/doc.php?id=7008378
https://winscp.net/eng/docs/faq_line_breaks#known_issues

ProFTPd handles it correctly and does not convert CRLF to CRCRLF.

try:
import sendfile
except ImportError:
sendfile = None
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was on purpose. I want tests to fail (and notice it) if pysendfile is not installed. (and make setup-dev-env will take care of that)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for explanation. Reverted my changes.

self.dummyfile.seek(0)
datafile = self.dummyfile.read()
self.assertEqual(len(data), len(datafile))
self.assertEqual(hash(data), hash(datafile))
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this test fail before applying the patch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does.

@giampaolo
Copy link
Owner

Could you please update HISTORY file describing the problem? That aside this LGTM.

@giampaolo
Copy link
Owner

...and CREDITS (add your name)

@andrewshulgin
Copy link
Contributor Author

Should I bump the version up to 1.5.5 in HISTORY.rst?

@andrewshulgin
Copy link
Contributor Author

Also, should I create an issue here on GitHub describing the problem in order to conform the HISTORY.rst format (#issue reference) or may I just refer to this PR?

@giampaolo giampaolo merged commit 3a5dfd3 into giampaolo:master Mar 27, 2019
@giampaolo
Copy link
Owner

Thanks.

@andrewshulgin
Copy link
Contributor Author

Thank you!

@andrewshulgin andrewshulgin deleted the ascii_crlf_fix branch March 27, 2019 11:34
@lurch
Copy link
Contributor

lurch commented Mar 27, 2019

Given that the code no longer uses os.linesep, should _posix_ascii_data_wrapper also be converting "\r not followed by \n" into "\r\n" for those OSes where os.linesep is \r ?

@andrewshulgin
Copy link
Contributor Author

andrewshulgin commented Mar 27, 2019

Regarding os.linesep == '\r' support I agree with you, I didn't take this into account, but AFAIK only MacOS 9 and lower use this separator; the latest Python version for it is 2.3. Am I correct? Does pyftpdlib support it?

Also, I believe we should not rely on os.linesep at all, and try to convert LF endings to CRLF even when os.linesep == '\r\n' but I tried to keep modifications as small as possible.

@giampaolo
Copy link
Owner

We support from 2.6 onwards. I'm not sure if there are systems other than OSX9 where os.sep == '\r'.

@lurch
Copy link
Contributor

lurch commented Mar 27, 2019

Just to clarify - I don't need support for \r line endings, I was just pointing it out in case it was something that got overlooked... 😉
(i.e. I won't be upset if support doesn't get added)

@giampaolo
Copy link
Owner

giampaolo commented Mar 27, 2019 via email

@andrewshulgin
Copy link
Contributor Author

andrewshulgin commented Mar 27, 2019

I've added CR to CRLF converion support in #494.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants