Bash cells show no intermediate output

[MarkDunne]

The %%bash cell does not show any output until everything is finished running.

For example

%%bash
echo "Hello 1"
sleep 5
echo "Hello 2"

Will not output "Hello 1" until the whole cell is done, 5 seconds later.

My use case for this is that in my pipeline, I have some rather long process calls that ideally I would like to run via my notebook. However, not being able to inspect the output while it's running makes things difficult. Clearly for python cells this isn't a problem, can we solve it for bash cells?

[kscd]

To me it seems that this is a bug in Ipython rather than in the notebook, since I was able to reproduce the same behavior in a simple ipython shell with the command

In [1]: %%bash
   ...: echo '1'
   ...: sleep 3
   ...: echo '2'
   ...: 

whereas

In [2]: import time
In [3]: print('1') ; time.sleep(3) ; print('2')

prints first the 1 and three seconds later the 2.

[tijlk]

I'm having the same problem. Would be nice to have this fixed. Otherwise I'm forced to run the bash scripts outside the notebook in order to have any idea of what's going on.

[mpacer]

@minrk is there anything we can do in terms of treating different lines of a bash cell (that aren't continued with\) as different messages so they can each send back their results piece by piece?

[minrk]

It doesn't need to be executed as a separate cell, we just need to make sure we flush and republish the output we receive from the subprocess more frequently. I don't recall how exactly we are doing that right now.

[ghost]

Would be great fix this for long running processes. Maybe something similar to what running bash scripts like !foo.sh does? The ! shortcut displays intermediate output. Thanks!

[ghost]

The solution seems to be not using the default python kernel but using the bash kernel from https://github.com/takluyver/bash_kernel

python -m bash_kernel.install

Restart your jupyter notebook, create a bash session, and you'll get immediate stdout/stderr display.

[boshokor]

this is rather a hack but it is possible to do it using subprocess module. following this question and assuming test.sh have the example given by @kscd it is possible to do it like this

import subprocess, time, os, sys
cmd = ["./test.sh"]

p = subprocess.Popen(cmd,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT)

for line in iter(p.stdout.readline, b''):
    print(">>> " + line.decode().rstrip())

[benlau]

I have made a cell magic called livebash to provide an alternative solution (plus extra features) for this problem:

https://github.com/benlau/ipylivebash