Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make redirection in a stream fashion #29

Closed
parafish opened this issue Mar 16, 2015 · 5 comments
Closed

Make redirection in a stream fashion #29

parafish opened this issue Mar 16, 2015 · 5 comments

Comments

@parafish
Copy link

Hello,

I am trying out this fancy language detection script,
and I am actually thinking -
do you think it makes more sense to make
for line in sys.stdin.readlines():
into
for line in sys.stdin:

in langid.py ?

@saffsd
Copy link
Owner

saffsd commented Mar 16, 2015

Hi Zheyi

Thanks for the suggestion. Does this change the behavior of the program,
especially across versions of Python? Or is the default iterator over
sys.stdin actually sys.stdin.readlines throughout? I haven't actually been
able to find the relevant Python documentation.

Cheers,
Marco

On Tue, Mar 17, 2015 at 2:48 AM, Rong Zheyi notifications@github.com
wrote:

Hello,

I am trying out this fancy language detection script,
and I am actually thinking -
do you think it makes more sense to make
for line in sys.stdin.readlines():
into
for line in sys.stdin:

in langid.py ?


Reply to this email directly or view it on GitHub
#29.

@parafish
Copy link
Author

I am not sure whether this would change the behavior of the program across versions of Python.
I raised this because readlines() will return a list, and this causes scalability problem - when the input is too big.

@jnothman
Copy link
Contributor

jnothman commented Sep 8, 2015

fwiw, for line in sys.stdin: should be functionally identically to for line in sys.stdin.readlines(): but streaming.

@jnothman
Copy link
Contributor

jnothman commented Sep 8, 2015

And I can't speak for anything before Python 2.3, but it worked then, I'm fairly certain.

tripleee pushed a commit to tripleee/langid.py that referenced this issue Sep 18, 2015
When classifying a line at a time, only read a line at a time, so as to
prevent Python from loading a large file into memory needlessly.
saffsd added a commit that referenced this issue Nov 24, 2015
@saffsd
Copy link
Owner

saffsd commented Nov 24, 2015

Fixed, thanks to tripleee, rongzhenyi and jnothman.

@saffsd saffsd closed this as completed Nov 24, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants