Skip to content
This repository has been archived by the owner on Jun 28, 2022. It is now read-only.

stdin should block and wait for new inputs till stdin is closed. #57

Merged
merged 1 commit into from
May 9, 2016

Conversation

ikedam
Copy link
Contributor

@ikedam ikedam commented Dec 8, 2015

StdinStreamRunner considers it reaches EOF when the stdin is not ready.
This results stream2es finishes before the previous program finishes the stream, especially when the program is slow.

This change blocks the runner while reading stdin, and considers reaching EOF only after stdin is closed.

Steps to reproduce problems:

  1. Create a script testoutput.sh:

    #!/bin/sh
    
    for i in $(seq 10); do
        TIMESTAMP=`date +%Y-%m-%dT%H:%M:%SZ`
        printf '{"@timestamp": "%s", "duration": 10}\n' "${TIMESTAMP}"
        sleep 1 # demonstrating a really slow program.
    done
    
  2. run stream2es:
    $ ./testoutput.sh |./stream2es stdin

Result (actual, unexpected): stream2es finishes before processing all the outputs from the script like this (only 3 lines are processed):

2015-12-08T23:10:54.503+0900 INFO  00:00.172 17.4d/s 1.5K/s (0.0mb) indexed 3 streamed 3 errors 0
2015-12-08T23:10:54.503+0900 INFO  done

Expected result (result after this change): all 10 lines are processed:

2015-12-08T23:12:11.330+0900 INFO  00:08.099 1.2d/s 0.1K/s (0.0mb) indexed 10 streamed 10 errors 0
2015-12-08T23:12:11.345+0900 INFO  done

@drewr drewr merged commit 6015950 into elastic:master May 9, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants