Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue #14149 - Do not discard a perfectly good line at the end of the data burst #67

Closed
wants to merge 1 commit into from
Closed

Issue #14149 - Do not discard a perfectly good line at the end of the data burst #67

wants to merge 1 commit into from

Conversation

rgov
Copy link

@rgov rgov commented Dec 28, 2018

Fixes #66.

While parsing a data burst, the inner loop consumes a line at a time and tests
if it matches the sensor data format. If it does not, the data burst is over,
so the new particle is emitted, and the function jumps back to the top o the
outer loop to start looking for the next data burst.

However, that line has now been consumed and cannot be read again by the outer
loop. If the discarded line is the header of the next data burst, parsing will
fail.

This commit fixes that by passing the line back to the outer loop.

@rgov
Do not discard a perfectly good line at the end of the data burst
a74ab00 
While parsing a data burst, the inner loop consumes a line at a time and tests
if it matches the sensor data format. If it does not, the data burst is over,
so the new particle is emitted, and the function jumps back to the top o the
outer loop to start looking for the next data burst.

However, that line has now been consumed and cannot be read again by the outer
loop. If the discarded line is the header of the next data burst, parsing will
fail.

This commit fixes that by passing the line back to the outer loop.
@rgov
Copy link
Author

rgov commented Jan 3, 2019

Test cases fail.

======================================================================
FAIL: test_parse_input (mi.dataset.parser.test.test_adcpt_acfgm_dcl_pd8.AdcptAcfgmPd8ParserUnitTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/oceanobservatories/mi-instrument/mi/dataset/parser/test/test_adcpt_acfgm_dcl_pd8.py", line 69, in test_parse_input
    self.assertListEqual(self.exception_callback_value, [])
AssertionError: Lists differ: [RecoverableSampleException(),... != []

First list contains 23 additional elements.
First extra element 0:
RecoverableSampleException()

Diff is 764 characters long. Set self.maxDiff to None to see it.
-------------------- >> begin captured logging << --------------------
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 00:04:23.320 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 01:04:20.834 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 02:04:22.337 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 03:04:21.573 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 04:04:21.357 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 05:04:20.218 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 06:04:22.217 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 07:04:20.135 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 08:04:23.119 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 09:04:23.103 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 10:04:23.078 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 11:04:19.721 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 12:04:20.915 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 13:04:23.008 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 14:04:19.884 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 15:04:23.909 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 16:04:20.864 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 17:04:21.009 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 19:04:20.907 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 20:04:20.013 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 21:04:20.085 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 22:04:22.620 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 23:04:20.015 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_recov (mi.dataset.parser.test.test_adcpt_acfgm_dcl_pd8.AdcptAcfgmPd8ParserUnitTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/oceanobservatories/mi-instrument/mi/dataset/parser/test/test_adcpt_acfgm_dcl_pd8.py", line 84, in test_recov
    self.assertListEqual(self.exception_callback_value, [])
AssertionError: Lists differ: [RecoverableSampleException(),... != []

First list contains 23 additional elements.
First extra element 0:
RecoverableSampleException()

Diff is 764 characters long. Set self.maxDiff to None to see it.
-------------------- >> begin captured logging << --------------------
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 00:04:23.320 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 01:04:20.834 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 02:04:22.337 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 03:04:21.573 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 04:04:21.357 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 05:04:20.218 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 06:04:22.217 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 07:04:20.135 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 08:04:23.119 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 09:04:23.103 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 10:04:23.078 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 11:04:19.721 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 12:04:20.915 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 13:04:23.008 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 14:04:19.884 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 15:04:23.909 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 16:04:20.864 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 17:04:21.009 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 19:04:20.907 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 20:04:20.013 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 21:04:20.085 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 22:04:22.620 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 23:04:20.015 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_telem (mi.dataset.parser.test.test_adcpt_acfgm_dcl_pd8.AdcptAcfgmPd8ParserUnitTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/oceanobservatories/mi-instrument/mi/dataset/parser/test/test_adcpt_acfgm_dcl_pd8.py", line 101, in test_telem
    self.assertListEqual(self.exception_callback_value, [])
AssertionError: Lists differ: [RecoverableSampleException(),... != []

First list contains 23 additional elements.
First extra element 0:
RecoverableSampleException()

Diff is 764 characters long. Set self.maxDiff to None to see it.
-------------------- >> begin captured logging << --------------------
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 00:04:23.320 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 01:04:20.834 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 02:04:22.337 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 03:04:21.573 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 04:04:21.357 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 05:04:20.218 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 06:04:22.217 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 07:04:20.135 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 08:04:23.119 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 09:04:23.103 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 10:04:23.078 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 11:04:19.721 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 12:04:20.915 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 13:04:23.008 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 14:04:19.884 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 15:04:23.909 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 16:04:20.864 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 17:04:21.009 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 19:04:20.907 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 20:04:20.013 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 21:04:20.085 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 22:04:22.620 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 23:04:20.015 \n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_telem_9692 (mi.dataset.parser.test.test_adcpt_acfgm_dcl_pd8.AdcptAcfgmPd8ParserUnitTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/oceanobservatories/mi-instrument/mi/dataset/parser/test/test_adcpt_acfgm_dcl_pd8.py", line 201, in test_telem_9692
    self.assertListEqual(self.exception_callback_value, [])
AssertionError: Lists differ: [RecoverableSampleException()] != []

First list contains 1 additional elements.
First extra element 0:
RecoverableSampleException()

- [RecoverableSampleException()]
+ []
-------------------- >> begin captured logging << --------------------
mi.dataset.parser.adcpt_acfgm_dcl_pd8: WARNING: Expected starting DCL Timestamp, received: '2013/12/01 00:04:23.320\n'
mi.dataset.test.test_parser: INFO: Received exception: RecoverableSampleException()
--------------------- >> end captured logging << ---------------------

@rgov rgov changed the title Do not discard a perfectly good line at the end of the data burst Issue #14149: Do not discard a perfectly good line at the end of the data burst Jan 14, 2019
@rgov rgov changed the title Issue #14149: Do not discard a perfectly good line at the end of the data burst Issue #14149 - Do not discard a perfectly good line at the end of the data burst Jan 14, 2019
@rgov rgov closed this by deleting the head repository Jul 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Line consumed by AdcptAcfgmPd8Parser needs to be put back if not used
1 participant
@rgov