File Source/Sink Dropping Samples

[ghostop14]

If you connect a file source block through a throttle to a file sink block and set the file source to not repeat, the resulting output file does not contain all of the samples in the original file. This was confirmed running GNURadio 3.7.13.4 on Ubuntu 16.04.

[fmagno]

My 2¢:

If Generate Options is set to QT GUI (illustrated above), the user is required to close the QT execution window for the remaining bytes to be dumped to the output file. It seems that the last chunk of data is buffered and it only gets flushed when the window is properly closed. I believe this is the undesired behaviour @ghostop14 is talking about.

However, if Generate Options is set to No GUI and Run Options is set to Run to Completion the issue does not occur (depicted below). The data is sent to the output and then the flowgraph stops executing automatically.

[ghostop14]

Interesting, I think I've probably also stopped it in the GNU Radio UI which I probably shouldn't do. My guess is that halted things more abruptly as well. Sounds like the scheduling engine may need one more check to work the last samples through?

[fmagno]

I would also like to raise another question for the core devs that I believe may be related to this one:

In this case, if I set the Generate Options to QT GUI and execute the flowgraph then no samples are lost if the execution window is closed properly.

However, this is not the case if Generate Options is set to No GUI (Run to Completion). Instead, the flowgraph remains running and requires an abrupt stop to cease the execution, implying loss of those samples that were left hanging in the buffer (issue originally raised by @ghostop14).

[noc0lour]

Both issues sound like a bug that needs fixing.

[777arc]

@fmagno I think the reason why your 2nd flowgraph example does not finish cleanly when the File Source is done, is because you have a block with a msg input, and there's no way for GNU Radio to know when that block is done getting an input (it doesn't realize that Tagged Steam to PDU converts a stream to msgs). So I think it makes sense that GNU Radio doesn't halt, because if you had a flowgraph that was just a bunch of blocks connected with msg ports (which is not uncommon), you wouldn't want GNU Radio to stop.

[777arc]

That being said, we really need to fix the issue that GNU Radio has always had, where the last few thousand samples from a File Source (or any block) get stuck in buffers unless you are using Run to Completion mode. Need a way to flush them through.

[bastibl]

Can you please test this again with the most recent master branch. I'm not able to reproduce this behavior and the discussion is getting a bit confusing.

Regarding the flow graph (file source -> throttle -> file sink):

• Until recently, the flow graph could only be shut down by closing the main window. CTRL-C in the terminal didn't work and GRC kill()ed the flow graph. This didn't allow any block to shutdown cleanly, i.e., the file sink was not able to close the handle. These issues should be fixed and I don't see any missing samples when closing the window, CTRL-C in the terminal, or stop it through GRC.
• There should be no problem with the No GUI versions.

Regarding the flow graph that converts to PDU in between:

• This flow graph should shut down cleanly with the 3.8 master branch. The QT version had the same issues as the other flow graph but they are fixed. What can go wrong is that PDUs can get lost when the message buffer of the PDUToTaggedStream block overflows. (We don't have back-pressure.)

Regarding run to completion with message ports:

• The TaggedStreamToPDU block should notice when its input buffer is done. After it processed the remaining samples it sends a done message to the system ports of its downstream blocks, which allows them to shut down. This is also part of QA tests:
  https://github.com/gnuradio/gnuradio/blob/master/gr-blocks/python/blocks/qa_pdu.py#L121-L128

Regarding stalled samples:

• Is there an example for this issue? The only thing I could imagine is if there is a block like TaggedStreamToPDU that requires a given number of samples to proceed. If the upstream block does not produce enough samples and does not terminate, the samples will stall in the buffer. But I think that's on purpose.

[fmagno]

Summary: I managed to test this with the gnuradio-next macports installation and I can still see an issue when stopping the execution with the GRC stop button. The good news is that right now the second case works well with the option Run to completion.

GR version: v3.8-MacPorts-next-git-8a7214c7(20190521-8a7214c7) (Python 2.7.16)

Legend:

• [OK] - all samples go through
• [FAIL] - remaining samples don't go through to the output file

File Source -> Throttle -> File Sink

• Options: Generate Options = QT GUI

  • Closing the window [OK]
  • Pressing GRC stop button [FAIL]

• Options: Generate Options = No GUI - Run to Completion [OK]

File Source -> S to TS -> TS to PDU -> PDU to TS -> File Sink

• Options: Generate Options = QT GUI

  • Closing the window [OK]
  • Pressing GRC stop button [FAIL]

• Options: Generate Options = No GUI - Run to Completion [OK]

[bastibl]

As far as I see, the build you are using does not include the commits that are supposed to fix this. It would be great if you could test it with a more recent version.

[fmagno]

Tested with 3.8.0.0-rc1 (Python 3.6.8) and it is fixed :)
Are we planning on backporting this fix to 3.7?

[devnulling]

@bastibl do you remember what commits were suppose to fix this? If we can, it'd make sense to back port to 3.7, but if not, we should close out this issue.

[nickoe]

@devnulling @bastibl I tried to see if I could find this, and I have a candidate without understanding the issue fully, but just reading the discussions briefly, that candidate is: #2806

This has been backported and merged to both maint-3.7 and maint-3.8.

@fmagno could you re-test it on maint-3.7 if that is what you are running? It looks like it is included in 3.7.14.0 and not fixed in 3.7.13.5, if the pull request I mentioned is the fix.

[michaelld]

For the case where there's a MSG between 2 streaming sections: It's not a huge surprise that one has to force quit, since the DONE signal from the source doesn't get propagated through the MSG connection, and hence the sink never receives it. This is a weakness of GR: no control channel that's regardless of the connection type. Fixing this case will not be trivial or easy, though it would be great to see it happen for some next GR version!

[michaelld]

For the case where there's a MSG between 2 streaming sections: It's not a huge surprise that one has to force quit, since the DONE signal from the source doesn't get propagated through the MSG connection, and hence the sink never receives it. This is a weakness of GR: no control channel that's regardless of the connection type. Fixing this case will not be trivial or easy, though it would be great to see it happen for some next GR version!

Or ... at least I think that's the way this works ... have to look at the code to be fully 100% sure!

[nickoe]

I just tested 3.8.0.0 and I see this for the three test cases:

Options: Generate Options = QT GUI
For both point 1. and 2. it seems to flush on the close or stop.

1. Closing the window [OK]
2. Pressing GRC stop button [OK]

Options: Generate Options = No GUI - Run to Completion

3. Run and wait [OK]

I see this as an indicator that the issue is likely fixed.

[nickoe]

Closing as based on my test above and earlier comment.
EDIT: Could also be related to the fix 0cc238f#diff-029198c1164bbb1fc60ed5ac394a415d

[Shruthi-Venkata-Sai-Shruthi]

How to use these methods to execute a flowgraph (attached in the image below), for a data file?

[michaelld]

How to use these methods to execute a flowgraph (attached in the image below), for a data file?

@Shruthi-Venkata-Sai-Shruthi you are better off posting your query to the GNU Radio users email group. This issue was fixed a long time ago & posting extra here will not result in more discussion. If you believe the issue was not fixed, then reopen the issue & provide specific evidence, and then folks will I'm sure jump in to debug / assist.