Skip to content

Failed tests don't print in the progress section, but appear in the results #705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
njtierney opened this issue Jan 17, 2018 · 4 comments · Fixed by #738
Closed

Failed tests don't print in the progress section, but appear in the results #705

njtierney opened this issue Jan 17, 2018 · 4 comments · Fixed by #738
Labels
bug an unexpected problem or unintended behavior

Comments

@njtierney
Copy link

njtierney commented Jan 17, 2018 •
edited
Loading

Hi there!

At this commit of naniar I get the following print message when I run devtools::test()

(apologies for hard coding the pkg PATH in the reprex, I couldn't get it to work otherwise!

The problem is that it doesn't print that I have 5 tests that failed, but shows me that I have 5 tests that failed.

When I run devtools::check() I can identify that the tests all fail in test-replace-with-na-where.

Perhaps I've missed something obvious, but thought you might like to know about this,

Thanks again for building and maintaining a package I use every day!

devtools::test("/Users/ntie0001/github/njtierney/naniar")
#> Loading naniar
#> Loading required package: testthat
#> Testing naniar
#> Warning: `encoding` is deprecated; all files now assumed to be UTF-8
#> ✔ | OK F W S | Context
#> 
⠏ |  0       | add_any_miss
⠋ |  1       | add_any_miss
⠙ |  2       | add_any_miss
⠹ |  3       | add_any_miss
⠸ |  4       | add_any_miss
⠼ |  5       | add_any_miss
⠴ |  6       | add_any_miss
⠦ |  7       | add_any_miss
✔ |  7       | add_any_miss [0.3 s]
#> 
⠏ |  0       | add_label_missings
⠋ |  1       | add_label_missings
⠙ |  2       | add_label_missings
⠹ |  3       | add_label_missings
✔ |  3       | add_label_missings
#> 
⠏ |  0       | add_label_shadow
⠋ |  1       | add_label_shadow
⠙ |  2       | add_label_shadow
⠹ |  3       | add_label_shadow
✔ |  3       | add_label_shadow
#> 
⠏ |  0       | add_n_miss
⠋ |  1       | add_n_miss
⠙ |  2       | add_n_miss
⠹ |  3       | add_n_miss
⠸ |  4       | add_n_miss
✔ |  4       | add_n_miss
#> 
⠏ |  0       | add_prop_miss
⠋ |  1       | add_prop_miss
⠙ |  2       | add_prop_miss
⠹ |  3       | add_prop_miss
⠸ |  4       | add_prop_miss
✔ |  4       | add_prop_miss
#> 
⠏ |  0       | add_shadow_shift
⠋ |  1       | add_shadow_shift
⠙ |  2       | add_shadow_shift
⠹ |  3       | add_shadow_shift
⠸ |  4       | add_shadow_shift
⠼ |  5       | add_shadow_shift
✔ |  5       | add_shadow_shift
#> 
⠏ |  0       | add_shadow
⠋ |  1       | add_shadow
⠙ |  2       | add_shadow
⠹ |  3       | add_shadow
⠸ |  4       | add_shadow
✔ |  4       | add_shadow
#> 
⠏ |  0       | as_shadow
⠋ |  1       | as_shadow
⠙ |  2       | as_shadow
⠹ |  3       | as_shadow
⠸ |  4       | as_shadow
⠼ |  5       | as_shadow
⠴ |  6       | as_shadow
⠦ |  7       | as_shadow
⠧ |  8       | as_shadow
⠇ |  9       | as_shadow
✔ |  9       | as_shadow
#> 
⠏ |  0       | bind_shadow
⠋ |  1       | bind_shadow
⠙ |  2       | bind_shadow
⠹ |  3       | bind_shadow
⠸ |  4       | bind_shadow
⠼ |  5       | bind_shadow
⠴ |  6       | bind_shadow
⠦ |  7       | bind_shadow
⠧ |  8       | bind_shadow
⠇ |  9       | bind_shadow
✔ |  9       | bind_shadow
#> 
⠏ |  0       | cast_shadow_shift_label
⠋ |  1       | cast_shadow_shift_label
⠙ |  2       | cast_shadow_shift_label
⠹ |  3       | cast_shadow_shift_label
⠸ |  4       | cast_shadow_shift_label
⠼ |  5       | cast_shadow_shift_label
⠴ |  6       | cast_shadow_shift_label
⠦ |  7       | cast_shadow_shift_label
⠧ |  8       | cast_shadow_shift_label
✔ |  8       | cast_shadow_shift_label [0.2 s]
#> 
⠏ |  0       | cast_shadow_shift
⠋ |  1       | cast_shadow_shift
⠙ |  2       | cast_shadow_shift
⠹ |  3       | cast_shadow_shift
⠸ |  4       | cast_shadow_shift
⠼ |  5       | cast_shadow_shift
⠴ |  6       | cast_shadow_shift
⠦ |  7       | cast_shadow_shift
⠧ |  8       | cast_shadow_shift
✔ |  8       | cast_shadow_shift [0.2 s]
#> 
⠏ |  0       | cast_shadow
⠋ |  1       | cast_shadow
⠙ |  2       | cast_shadow
⠹ |  3       | cast_shadow
⠸ |  4       | cast_shadow
⠼ |  5       | cast_shadow
⠴ |  6       | cast_shadow
⠦ |  7       | cast_shadow
✔ |  7       | cast_shadow [0.1 s]
#> 
⠏ |  0       | gather_shadow
⠋ |  1       | gather_shadow
⠙ |  2       | gather_shadow
⠹ |  3       | gather_shadow
⠸ |  4       | gather_shadow
⠼ |  5       | gather_shadow
⠴ |  6       | gather_shadow
⠦ |  7       | gather_shadow
⠧ |  8       | gather_shadow
⠇ |  9       | gather_shadow
✔ |  9       | gather_shadow [0.1 s]
#> 
⠏ |  0       | Label 1d missings
⠋ |  1       | Label 1d missings
⠙ |  2       | Label 1d missings
⠹ |  3       | Label 1d missings
⠸ |  4       | Label 1d missings
⠼ |  5       | Label 1d missings
⠴ |  6       | Label 1d missings
✔ |  6       | Label 1d missings
#> 
⠏ |  0       | Label 2d missings
⠋ |  1       | Label 2d missings
⠙ |  2       | Label 2d missings
⠹ |  3       | Label 2d missings
⠸ |  4       | Label 2d missings
⠼ |  5       | Label 2d missings
⠴ |  6       | Label 2d missings
⠦ |  7       | Label 2d missings
⠧ |  8       | Label 2d missings
✔ |  8       | Label 2d missings
#> 
⠏ |  0       | label_shadow_matrix behaves
⠋ |  1       | label_shadow_matrix behaves
⠙ |  2       | label_shadow_matrix behaves
⠹ |  3       | label_shadow_matrix behaves
⠸ |  4       | label_shadow_matrix behaves
✔ |  4       | label_shadow_matrix behaves
#> 
⠏ |  0       | label_shadow
⠋ |  1       | label_shadow
⠙ |  2       | label_shadow
⠹ |  3       | label_shadow
✔ |  3       | label_shadow
#> 
⠏ |  0       | miss_case_summary tidiers
⠋ |  1       | miss_case_summary tidiers
⠙ |  2       | miss_case_summary tidiers
⠹ |  3       | miss_case_summary tidiers
⠸ |  4       | miss_case_summary tidiers
⠼ |  5       | miss_case_summary tidiers
⠴ |  6       | miss_case_summary tidiers
⠦ |  7       | miss_case_summary tidiers
⠧ |  8       | miss_case_summary tidiers
⠇ |  9       | miss_case_summary tidiers
✔ |  9       | miss_case_summary tidiers [0.4 s]
#> 
⠏ |  0       | miss_case_table tidiers
⠋ |  1       | miss_case_table tidiers
⠙ |  2       | miss_case_table tidiers
⠹ |  3       | miss_case_table tidiers
⠸ |  4       | miss_case_table tidiers
⠼ |  5       | miss_case_table tidiers
⠴ |  6       | miss_case_table tidiers
⠦ |  7       | miss_case_table tidiers
⠧ |  8       | miss_case_table tidiers
⠇ |  9       | miss_case_table tidiers
⠏ | 10       | miss_case_table tidiers
✔ | 10       | miss_case_table tidiers [0.3 s]
#> 
⠏ |  0       | miss_prop_summary
⠋ |  1       | miss_prop_summary
⠙ |  2       | miss_prop_summary
⠹ |  3       | miss_prop_summary
⠸ |  4       | miss_prop_summary
⠼ |  5       | miss_prop_summary
⠴ |  6       | miss_prop_summary
⠦ |  7       | miss_prop_summary
⠧ |  8       | miss_prop_summary
⠇ |  9       | miss_prop_summary
⠏ | 10       | miss_prop_summary
⠋ | 11       | miss_prop_summary
⠙ | 12       | miss_prop_summary
⠹ | 13       | miss_prop_summary
✔ | 13       | miss_prop_summary [0.4 s]
#> 
⠏ |  0       | miss_summary
⠋ |  1       | miss_summary
⠙ |  2       | miss_summary
⠹ |  3       | miss_summary
⠸ |  4       | miss_summary
⠼ |  5       | miss_summary
⠴ |  6       | miss_summary
⠦ |  7       | miss_summary
⠧ |  8       | miss_summary
✔ |  8       | miss_summary [0.2 s]
#> 
⠏ |  0       | miss_var_summary tidiers
⠋ |  1       | miss_var_summary tidiers
⠙ |  2       | miss_var_summary tidiers
⠹ |  3       | miss_var_summary tidiers
⠸ |  4       | miss_var_summary tidiers
⠼ |  5       | miss_var_summary tidiers
⠴ |  6       | miss_var_summary tidiers
⠦ |  7       | miss_var_summary tidiers
⠧ |  8       | miss_var_summary tidiers
⠇ |  9       | miss_var_summary tidiers
⠏ | 10       | miss_var_summary tidiers
✔ | 10       | miss_var_summary tidiers [0.3 s]
#> 
⠏ |  0       | miss_var_table tidiers
⠋ |  1       | miss_var_table tidiers
⠙ |  2       | miss_var_table tidiers
⠹ |  3       | miss_var_table tidiers
⠸ |  4       | miss_var_table tidiers
⠼ |  5       | miss_var_table tidiers
⠴ |  6       | miss_var_table tidiers
⠦ |  7       | miss_var_table tidiers
⠧ |  8       | miss_var_table tidiers
⠇ |  9       | miss_var_table tidiers
⠏ | 10       | miss_var_table tidiers
✔ | 10       | miss_var_table tidiers [0.4 s]
#> 
⠏ |  0       | miss_*_pct tidiers
⠋ |  1       | miss_*_pct tidiers
⠙ |  2       | miss_*_pct tidiers
⠹ |  3       | miss_*_pct tidiers
⠸ |  4       | miss_*_pct tidiers
⠼ |  5       | miss_*_pct tidiers
⠴ |  6       | miss_*_pct tidiers
⠦ |  7       | miss_*_pct tidiers
⠧ |  8       | miss_*_pct tidiers
⠇ |  9       | miss_*_pct tidiers
⠏ | 10       | miss_*_pct tidiers
⠋ | 11       | miss_*_pct tidiers
⠙ | 12       | miss_*_pct tidiers
⠹ | 13       | miss_*_pct tidiers
⠸ | 14       | miss_*_pct tidiers
⠼ | 15       | miss_*_pct tidiers
✔ | 15       | miss_*_pct tidiers
#> 
⠏ |  0       | n_complete
⠋ |  1       | n_complete
⠙ |  2       | n_complete
⠹ |  3       | n_complete
⠸ |  4       | n_complete
⠼ |  5       | n_complete
⠴ |  6       | n_complete
⠦ |  7       | n_complete
✔ |  7       | n_complete
#> 
⠏ |  0       | n_miss
⠋ |  1       | n_miss
⠙ |  2       | n_miss
⠹ |  3       | n_miss
⠸ |  4       | n_miss
⠼ |  5       | n_miss
⠴ |  6       | n_miss
⠦ |  7       | n_miss
✔ |  7       | n_miss
#> 
⠏ |  0       | naniar-plots
#> 
⠋ |  1       | naniar-plots
#> 
⠙ |  2       | naniar-plots
#> 
⠹ |  3       | naniar-plots
#> 
⠸ |  4       | naniar-plots
#> 
⠼ |  5       | naniar-plots
#> 
⠴ |  6       | naniar-plots
✔ |  6       | naniar-plots [2.8 s]
#> 
⠏ |  0       | replace_with_na scoped variants
⠋ |  1       | replace_with_na scoped variants
⠙ |  2       | replace_with_na scoped variants
⠹ |  3       | replace_with_na scoped variants
⠸ |  4       | replace_with_na scoped variants
⠼ |  5       | replace_with_na scoped variants
⠴ |  6       | replace_with_na scoped variants
⠦ |  7       | replace_with_na scoped variants
⠧ |  8       | replace_with_na scoped variants
⠇ |  9       | replace_with_na scoped variants
⠏ | 10       | replace_with_na scoped variants
⠋ | 11       | replace_with_na scoped variants
✔ | 11       | replace_with_na scoped variants
#> 
⠙ | 12       | replace_with_na scoped variants
⠹ | 13       | replace_with_na scoped variants
⠸ | 13 1     | replace_with_na scoped variants
⠼ | 14 1     | replace_with_na scoped variants
⠴ | 15 1     | replace_with_na scoped variants
⠦ | 15 2     | replace_with_na scoped variants
⠧ | 16 2     | replace_with_na scoped variants
⠇ | 16 3     | replace_with_na scoped variants
⠏ | 16 4     | replace_with_na scoped variants
⠋ | 17 4     | replace_with_na scoped variants
⠙ | 17 5     | replace_with_na scoped variants
⠏ |  0       | replace_to_na deprecation test
⠋ |  1       | replace_to_na deprecation test
✔ |  1       | replace_to_na deprecation test
#> 
⠏ |  0       | replace_with_na
⠋ |  1       | replace_with_na
⠙ |  2       | replace_with_na
⠹ |  3       | replace_with_na
⠸ |  4       | replace_with_na
⠼ |  5       | replace_with_na
⠴ |  6       | replace_with_na
⠦ |  7       | replace_with_na
⠧ |  8       | replace_with_na
✔ |  8       | replace_with_na
#> 
⠏ |  0       | shadow shift
⠋ |  1       | shadow shift
⠙ |  2       | shadow shift
⠹ |  3       | shadow shift
⠸ |  4       | shadow shift
⠼ |  5       | shadow shift
⠴ |  6       | shadow shift
⠦ |  7       | shadow shift
⠧ |  8       | shadow shift
⠇ |  9       | shadow shift
✔ |  9       | shadow shift
#> 
#> ══ Results ════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
#> Duration: 6.6 s
#> 
#> OK:       231
#> Failed:   5
#> Warnings: 0
#> Skipped:  0
#> 
#> No-one gets it right on their first try
Session info 
devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.4.3 (2017-11-30)
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_AU.UTF-8                 
#>  tz       Australia/Melbourne         
#>  date     2018-01-17
#> Packages -----------------------------------------------------------------
#>  package           * version    date      
#>  assertthat          0.2.0      2017-04-11
#>  backports           1.1.2      2017-12-13
#>  base              * 3.4.3      2017-12-07
#>  bindr               0.1        2016-11-13
#>  bindrcpp          * 0.2        2017-06-17
#>  cli                 1.0.0      2017-11-05
#>  colorspace          1.3-2      2016-12-14
#>  commonmark          1.4        2017-09-01
#>  compiler            3.4.3      2017-12-07
#>  crayon              1.3.4      2017-10-09
#>  datasets          * 3.4.3      2017-12-07
#>  desc                1.1.1      2017-08-03
#>  devtools            1.13.4     2017-11-09
#>  digest              0.6.14     2018-01-14
#>  dplyr               0.7.4      2017-09-28
#>  evaluate            0.10.1     2017-06-24
#>  fontBitstreamVera   0.1.1      2017-02-01
#>  fontLiberation      0.1.0      2016-10-15
#>  fontquiver          0.2.1      2017-02-01
#>  forcats             0.2.0      2017-01-23
#>  formatR             1.5        2017-04-25
#>  gdtools           * 0.1.6      2017-09-01
#>  ggplot2           * 2.2.1.9000 2017-12-06
#>  glue                1.2.0      2017-10-29
#>  graphics          * 3.4.3      2017-12-07
#>  grDevices         * 3.4.3      2017-12-07
#>  grid                3.4.3      2017-12-07
#>  gridExtra           2.3        2017-09-09
#>  gtable              0.2.0      2016-02-26
#>  htmltools           0.3.6      2017-04-28
#>  knitr               1.17       2017-08-10
#>  labeling            0.3        2014-08-23
#>  lazyeval            0.2.1      2017-10-29
#>  magrittr            1.5        2014-11-22
#>  memoise             1.1.0      2017-04-21
#>  methods           * 3.4.3      2017-12-07
#>  munsell             0.4.3      2016-02-13
#>  naniar            * 0.1.4.9100 <NA>      
#>  pillar              1.1.0      2018-01-14
#>  pkgconfig           2.0.1      2017-03-21
#>  plyr                1.8.4      2016-06-08
#>  purrr               0.2.4      2017-10-18
#>  purrrlyr            0.0.2      2017-05-13
#>  R6                  2.2.2      2017-06-17
#>  Rcpp                0.12.14    2017-11-23
#>  rlang               0.1.6      2017-12-21
#>  rmarkdown           1.8        2017-11-17
#>  roxygen2            6.0.1.9000 2017-05-10
#>  rprojroot           1.3-1      2017-12-18
#>  scales              0.5.0.9000 2017-12-06
#>  stats             * 3.4.3      2017-12-07
#>  stringi             1.1.6      2017-11-17
#>  stringr             1.2.0      2017-02-18
#>  svglite             1.2.1      2017-09-11
#>  testthat          * 2.0.0      2017-12-13
#>  tibble              1.4.1      2017-12-25
#>  tidyr               0.7.2      2017-10-16
#>  tidyselect          0.2.3      2017-11-06
#>  tools               3.4.3      2017-12-07
#>  utils             * 3.4.3      2017-12-07
#>  vdiffr              0.2.1      2017-10-09
#>  viridis             0.4.0      2017-03-27
#>  viridisLite         0.2.0      2017-03-24
#>  visdat              0.2.2      2017-12-14
#>  withr               2.1.1      2017-12-19
#>  xml2                1.1.1      2017-01-24
#>  yaml                2.1.16     2017-12-12
#>  source                             
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.3)                     
#>  local                              
#>  cran (@0.1)                        
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.1)                     
#>  local                              
#>  Github (gaborcsardi/crayon@b5221ab)
#>  local                              
#>  CRAN (R 3.4.1)                     
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.2)                     
#>  cran (@0.10.1)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.1)                     
#>  Github (tidyverse/ggplot2@7b5c185) 
#>  cran (@1.2.0)                      
#>  local                              
#>  local                              
#>  local                              
#>  CRAN (R 3.4.1)                     
#>  CRAN (R 3.4.0)                     
#>  cran (@0.3.6)                      
#>  cran (@1.17)                       
#>  CRAN (R 3.4.0)                     
#>  cran (@0.2.1)                      
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  local                              
#>  CRAN (R 3.4.0)                     
#>  local                              
#>  CRAN (R 3.4.3)                     
#>  cran (@2.0.1)                      
#>  CRAN (R 3.4.0)                     
#>  cran (@0.2.4)                      
#>  CRAN (R 3.4.0)                     
#>  cran (@2.2.2)                      
#>  cran (@0.12.14)                    
#>  cran (@0.1.6)                      
#>  CRAN (R 3.4.2)                     
#>  Github (klutometis/roxygen@14eae41)
#>  CRAN (R 3.4.3)                     
#>  Github (hadley/scales@d767915)     
#>  local                              
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.1)                     
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.2)                     
#>  local                              
#>  local                              
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.0)                     
#>  cran (@0.2.0)                      
#>  local                              
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.3)
@hadley
Copy link
Member

hadley commented Jan 24, 2018

I wonder if it's related to having two contexts in https://github.com/njtierney/naniar/blob/master/tests/testthat/test-replace-with-na.R. It's possible there's a bug related to multiple contexts in one file.

@hadley
Copy link
Member

hadley commented Jan 24, 2018

Do you still have the problem with a different reporter?

@njtierney
Copy link
Author

Ooop - sorry, I thought I wrote more about this - I didn't specify context() in the file, this was why it didn't list them in the new reporter, although it did list them in the reporter that runs when I do devtools::check().

@jimhester jimhester added the bug an unexpected problem or unintended behavior label Mar 29, 2018
@jimhester
Copy link
Member

marking this as a bug, as we should still be reporting test failures even if there is no context (by having a default context).

jimhester added a commit to jimhester/testthat that referenced this issue Apr 2, 2018
@jimhester
Generate a context if one does not exist
dff9aea 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter. This generates a
context from the filename if there is not already a defined context,
fixing the issue.

Fixes r-lib#700
Fixes r-lib#705
@jimhester jimhester mentioned this issue Apr 2, 2018
Merged
jimhester added a commit to jimhester/testthat that referenced this issue Apr 2, 2018
@jimhester
Generate a context if one does not exist
3dcc732 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter. This generates a
context from the filename if there is not already a defined context,
fixing the issue.

Fixes r-lib#700
Fixes r-lib#705
jimhester added a commit to jimhester/testthat that referenced this issue Apr 2, 2018
@jimhester
Generate a context if one does not exist
b85e069 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter and a different (unreported) error in
the JUnit reporter. This generates a context from the filename if there
is not already a defined context, fixing the issue.

Fixes r-lib#700
Fixes r-lib#705
jimhester added a commit that referenced this issue Apr 4, 2018
@jimhester
Generate a context if one does not exist (#738)
42e3a79 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter and a different (unreported) error in
the JUnit reporter. This generates a context from the filename if there
is not already a defined context, fixing the issue.

Fixes #700
Fixes #705
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug an unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Generate a context if one does not exist jimhester/testthat
3 participants
@hadley @jimhester @njtierney