read_lines(fname,skip=xxx) skipping empty lines causes unexpected, backward-incompatible behavior

[dan-reznik]

Let 'file.txt' contain the following 8 lines, with its 4th line empty:

L1
L2
L3
<empty line>
Hello,Weird,Behavior
1,2,3
3,4,5
6,7,8

Calling read_lines() with no skip parameter produces the expected results, with sk0's 4th element being the empty string:

library(tidyverse)
fname <- 'file.txt'
sk0 <- read_lines(fname)
sk0
#> "L1","L2","L3","","Hello,Weird,Behavior","1,2,3","3,4,5,"6,7,8"    

Skipping the first 3 lines also works, now with the 1st element being an empty string:

sk3 <- read_lines(fname,skip=3)
sk3
#> "","Hello,Weird,Behavior","1,2,3","3,4,5","6,7,8"               

However, and here lies the bug, skipping 4 lines misses the 5th line entirely! (in fact skip=4 will miss any sequence of empty lines).

sk4 <- read_lines(fname,skip=4)
sk4
#> "1,2,3","3,4,5","6,7,8"

Though skipping empty lines is a documented feature of this version of readr, it seems inconsistent and unnecessary, and will result in backward incompatibilities.

sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] data.table_1.11.8 snakecase_0.9.2   bindrcpp_0.2.2    assertthat_0.2.0  stringi_1.2.4    
 [6] jsonlite_1.5      lubridate_1.7.4   glue_1.3.0        readxl_1.1.0      forcats_0.3.0    
[11] stringr_1.3.1     dplyr_0.7.8       purrr_0.2.5       readr_1.2.1       tidyr_0.8.2      
[16] tibble_1.4.2      ggplot2_3.1.0     tidyverse_1.2.1  

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0       cellranger_1.1.0 pillar_1.3.0     compiler_3.5.1   plyr_1.8.4      
 [6] bindr_0.1.1      tools_3.5.1      nlme_3.1-137     gtable_0.2.0     lattice_0.20-38 
[11] pkgconfig_2.0.2  rlang_0.3.0.1    cli_1.0.1        rstudioapi_0.8   yaml_2.2.0      
[16] haven_2.0.0      withr_2.1.2      xml2_1.2.0       httr_1.3.1       hms_0.4.2       
[21] grid_3.5.1       tidyselect_0.2.5 R6_2.3.0         modelr_0.1.2     magrittr_1.5    
[26] backports_1.1.2  scales_1.0.0     rvest_0.3.2      colorspace_1.3-2 lazyeval_0.2.1  
[31] munsell_0.5.0    broom_0.5.0      crayon_1.3.4 

[jimhester]

This is the documented behavior, see https://github.com/tidyverse/readr/blob/master/NEWS.md#blank-line-skipping.

[dan-reznik]

Jim this behavior cannot be correct since there is no value of the skip parameter that will retrieve the first non blank line after the sequence of blanks (in my case line 5). Note also that skip=3 does not skip any blank lines (first retrieved value is "") however skip=4 skips the first non blank line. this is inconsistent. This "new" behavior should minimally be controllable via a parameter: skip_empty_lines=T/F Dan

…

On Fri, Nov 23, 2018, 1:51 PM Jim Hester ***@***.*** wrote: This is the documented behavior, see https://github.com/tidyverse/readr/blob/master/NEWS.md#blank-line-skipping . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#923 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APeqloFFngcX2HreOh0fcU-jT7e_ribbks5uyBmNgaJpZM4YwwFB> .

[zhangj5]

This appears to also cause an issue in "GEOquery". Downgrading to 1.1.1 fixed the problem.
seandavi/GEOquery#74

[rogiersbart]

Also, when programmatically determining the number of lines to skip, this requires additional checking of the number of empty lines, instead of just finding the line number you want to start at minus 1. It's also a question of semantics I guess: For me, an empty line is still a line. An additional argument seems great, and I'd love the default to remain like it was before (empty line == line), but that's just how I feel about it, and maybe there is good reason to change the default behaviour ...

[nacnudus]

This seems to have appeared in 6517708.

[seandavi]

Thanks, @zhangj5 for pulling in the GEOquery issue seandavi/GEOquery#74. This is a breaking change for GEOquery.

[jimhester]

@dan-reznik please refrain from commenting further unless you have more detail to add to the issue.

[nacnudus]

I think the problem is in the first phase of line-skipping from the top of the file to the start of the data.

readr/src/Source.h

Line 28 in d52a177

while (n > 0 && cur != end) {

n is the number of lines the user wants to skip. While n > 0 all lines are skipped including blank lines and comments, but once n == 0 any remaining blank lines and comments remain. This is usually disguised by read_delim() and read_table() because they do more phases of skipping to read column names and then data, but you can force some strange behaviour.

txt <- "note\n\n\ncol_name\ndata"
readr::read_lines(txt, skip = 1)
#> [1] ""         ""         "col_name" "data"
readr::read_csv(txt, skip = 1)
#> # A tibble: 1 x 1
#>   col_name
#>   <chr>   
#> 1 data
readr::read_csv(txt, skip = 1, skip_empty_rows = FALSE)
#> Warning: Missing column names filled in: 'X1' [1]
#> # A tibble: 1 x 1
#>   X1   
#>   <chr>
#> 1 data

Created on 2018-11-30 by the reprex package (v0.2.0.9000).

[seandavi]

@jimhester, when you get a chance, could you clarify the path forward? Should we be making changes to code to address the change, or is there a plan to revert the change? If there has been no decision made, is there a timeframe for doing so? Just looking for an update if you have one.

[jennybc]

I'll chime in, in case @jimhester has already entered the weekend zone.

Yes there's a bug and a fix is already in progress. But it requires some effort to fix this w/o breaking something else.

I advise you hold off on any sort of workaround on your end.

[seandavi]

Thx, @jennybc, for chiming in. I totally understand how fixes when dependent packages are involved can be a tightrope walk.

[pabgamu]

@jennybc guess you are already aware but this behaviour also affects read_tsv, using similar sample

txt <- "note\n\n\ncol_name\ndata\n"

txt
[1] "note\n\n\ncol_name\ndata\n"

read_tsv(txt, skip = 0, skip_empty_rows = FALSE, col_names = FALSE)
# A tibble: 5 x 1
  X1      
  <chr>   
1 note    
2 NA      
3 NA      
4 col_name
5 data    

read_tsv(txt, skip = 1, skip_empty_rows = FALSE, col_names = FALSE)
# A tibble: 4 x 1
  X1      
  <chr>   
1 NA      
2 NA      
3 col_name
4 data    

read_tsv(txt, skip = 2, skip_empty_rows = FALSE, col_names = FALSE)
# A tibble: 1 x 1
  X1   
  <chr>
1 data 

sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.1 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=es_ES.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=es_ES.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=es_ES.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=es_ES.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
 [1] reprex_0.2.1    bindrcpp_0.2.2  forcats_0.3.0   stringr_1.3.1  
 [5] dplyr_0.7.8     purrr_0.2.5     readr_1.2.1     tidyr_0.8.2    
 [9] tibble_1.4.2    ggplot2_3.1.0   tidyverse_1.2.1

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5 haven_2.0.0      lattice_0.20-38 
 [4] colorspace_1.3-2 htmltools_0.3.6  base64enc_0.1-3 
 [7] yaml_2.2.0       utf8_1.1.4       rlang_0.3.0.1   
[10] pillar_1.3.0     glue_1.3.0       withr_2.1.2     
[13] modelr_0.1.2     readxl_1.1.0     bindr_0.1.1     
[16] plyr_1.8.4       munsell_0.5.0    gtable_0.2.0    
[19] cellranger_1.1.0 rvest_0.3.2      evaluate_0.12   
[22] knitr_1.20       callr_3.0.0      ps_1.2.1        
[25] fansi_0.4.0      broom_0.5.0      Rcpp_1.0.0      
[28] clipr_0.4.1      scales_1.0.0     backports_1.1.2 
[31] debugme_1.1.0    jsonlite_1.5     fs_1.2.6        
[34] hms_0.4.2        digest_0.6.18    stringi_1.2.4   
[37] processx_3.2.0   grid_3.5.1       rprojroot_1.3-2 
[40] cli_1.0.1        tools_3.5.1      magrittr_1.5    
[43] lazyeval_0.2.1   whisker_0.3-2    crayon_1.3.4    
[46] pkgconfig_2.0.2  xml2_1.2.0       lubridate_1.7.4 
[49] assertthat_0.2.0 rmarkdown_1.10   httr_1.3.1      
[52] rstudioapi_0.8   R6_2.3.0         nlme_3.1-137    
[55] compiler_3.5.1  
#> Error: <text>:4:1: unexpected '['
#> 3: txt
#> 4: [
#>    ^

^{Created on 2018-12-04 by the reprex package (v0.2.1)}

[bknakker]

The fact that I'm unhappy with this breaking change is my issue, but please don't leave the parameter that can reverse this undocumented.