/testthat

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed tests don't print in the progress section, but appear in the results #705

Closed
njtierney opened this Issue Jan 17, 2018 · 4 comments

Comments

Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
3 participants
@njtierney @hadley @jimhester
@njtierney

njtierney commented Jan 17, 2018 •
edited

Hi there!

At this commit of naniar I get the following print message when I run devtools::test()

(apologies for hard coding the pkg PATH in the reprex, I couldn't get it to work otherwise!

The problem is that it doesn't print that I have 5 tests that failed, but shows me that I have 5 tests that failed.

When I run devtools::check() I can identify that the tests all fail in test-replace-with-na-where.

Perhaps I've missed something obvious, but thought you might like to know about this,

Thanks again for building and maintaining a package I use every day!

devtools::test("/Users/ntie0001/github/njtierney/naniar")
#> Loading naniar
#> Loading required package: testthat
#> Testing naniar
#> Warning: `encoding` is deprecated; all files now assumed to be UTF-8
#> ✔ | OK F W S | Context
#> 
⠏ |  0       | add_any_miss
⠋ |  1       | add_any_miss
⠙ |  2       | add_any_miss
⠹ |  3       | add_any_miss
⠸ |  4       | add_any_miss
⠼ |  5       | add_any_miss
⠴ |  6       | add_any_miss
⠦ |  7       | add_any_miss
✔ |  7       | add_any_miss [0.3 s]
#> 
⠏ |  0       | add_label_missings
⠋ |  1       | add_label_missings
⠙ |  2       | add_label_missings
⠹ |  3       | add_label_missings
✔ |  3       | add_label_missings
#> 
⠏ |  0       | add_label_shadow
⠋ |  1       | add_label_shadow
⠙ |  2       | add_label_shadow
⠹ |  3       | add_label_shadow
✔ |  3       | add_label_shadow
#> 
⠏ |  0       | add_n_miss
⠋ |  1       | add_n_miss
⠙ |  2       | add_n_miss
⠹ |  3       | add_n_miss
⠸ |  4       | add_n_miss
✔ |  4       | add_n_miss
#> 
⠏ |  0       | add_prop_miss
⠋ |  1       | add_prop_miss
⠙ |  2       | add_prop_miss
⠹ |  3       | add_prop_miss
⠸ |  4       | add_prop_miss
✔ |  4       | add_prop_miss
#> 
⠏ |  0       | add_shadow_shift
⠋ |  1       | add_shadow_shift
⠙ |  2       | add_shadow_shift
⠹ |  3       | add_shadow_shift
⠸ |  4       | add_shadow_shift
⠼ |  5       | add_shadow_shift
✔ |  5       | add_shadow_shift
#> 
⠏ |  0       | add_shadow
⠋ |  1       | add_shadow
⠙ |  2       | add_shadow
⠹ |  3       | add_shadow
⠸ |  4       | add_shadow
✔ |  4       | add_shadow
#> 
⠏ |  0       | as_shadow
⠋ |  1       | as_shadow
⠙ |  2       | as_shadow
⠹ |  3       | as_shadow
⠸ |  4       | as_shadow
⠼ |  5       | as_shadow
⠴ |  6       | as_shadow
⠦ |  7       | as_shadow
⠧ |  8       | as_shadow
⠇ |  9       | as_shadow
✔ |  9       | as_shadow
#> 
⠏ |  0       | bind_shadow
⠋ |  1       | bind_shadow
⠙ |  2       | bind_shadow
⠹ |  3       | bind_shadow
⠸ |  4       | bind_shadow
⠼ |  5       | bind_shadow
⠴ |  6       | bind_shadow
⠦ |  7       | bind_shadow
⠧ |  8       | bind_shadow
⠇ |  9       | bind_shadow
✔ |  9       | bind_shadow
#> 
⠏ |  0       | cast_shadow_shift_label
⠋ |  1       | cast_shadow_shift_label
⠙ |  2       | cast_shadow_shift_label
⠹ |  3       | cast_shadow_shift_label
⠸ |  4       | cast_shadow_shift_label
⠼ |  5       | cast_shadow_shift_label
⠴ |  6       | cast_shadow_shift_label
⠦ |  7       | cast_shadow_shift_label
⠧ |  8       | cast_shadow_shift_label
✔ |  8       | cast_shadow_shift_label [0.2 s]
#> 
⠏ |  0       | cast_shadow_shift
⠋ |  1       | cast_shadow_shift
⠙ |  2       | cast_shadow_shift
⠹ |  3       | cast_shadow_shift
⠸ |  4       | cast_shadow_shift
⠼ |  5       | cast_shadow_shift
⠴ |  6       | cast_shadow_shift
⠦ |  7       | cast_shadow_shift
⠧ |  8       | cast_shadow_shift
✔ |  8       | cast_shadow_shift [0.2 s]
#> 
⠏ |  0       | cast_shadow
⠋ |  1       | cast_shadow
⠙ |  2       | cast_shadow
⠹ |  3       | cast_shadow
⠸ |  4       | cast_shadow
⠼ |  5       | cast_shadow
⠴ |  6       | cast_shadow
⠦ |  7       | cast_shadow
✔ |  7       | cast_shadow [0.1 s]
#> 
⠏ |  0       | gather_shadow
⠋ |  1       | gather_shadow
⠙ |  2       | gather_shadow
⠹ |  3       | gather_shadow
⠸ |  4       | gather_shadow
⠼ |  5       | gather_shadow
⠴ |  6       | gather_shadow
⠦ |  7       | gather_shadow
⠧ |  8       | gather_shadow
⠇ |  9       | gather_shadow
✔ |  9       | gather_shadow [0.1 s]
#> 
⠏ |  0       | Label 1d missings
⠋ |  1       | Label 1d missings
⠙ |  2       | Label 1d missings
⠹ |  3       | Label 1d missings
⠸ |  4       | Label 1d missings
⠼ |  5       | Label 1d missings
⠴ |  6       | Label 1d missings
✔ |  6       | Label 1d missings
#> 
⠏ |  0       | Label 2d missings
⠋ |  1       | Label 2d missings
⠙ |  2       | Label 2d missings
⠹ |  3       | Label 2d missings
⠸ |  4       | Label 2d missings
⠼ |  5       | Label 2d missings
⠴ |  6       | Label 2d missings
⠦ |  7       | Label 2d missings
⠧ |  8       | Label 2d missings
✔ |  8       | Label 2d missings
#> 
⠏ |  0       | label_shadow_matrix behaves
⠋ |  1       | label_shadow_matrix behaves
⠙ |  2       | label_shadow_matrix behaves
⠹ |  3       | label_shadow_matrix behaves
⠸ |  4       | label_shadow_matrix behaves
✔ |  4       | label_shadow_matrix behaves
#> 
⠏ |  0       | label_shadow
⠋ |  1       | label_shadow
⠙ |  2       | label_shadow
⠹ |  3       | label_shadow
✔ |  3       | label_shadow
#> 
⠏ |  0       | miss_case_summary tidiers
⠋ |  1       | miss_case_summary tidiers
⠙ |  2       | miss_case_summary tidiers
⠹ |  3       | miss_case_summary tidiers
⠸ |  4       | miss_case_summary tidiers
⠼ |  5       | miss_case_summary tidiers
⠴ |  6       | miss_case_summary tidiers
⠦ |  7       | miss_case_summary tidiers
⠧ |  8       | miss_case_summary tidiers
⠇ |  9       | miss_case_summary tidiers
✔ |  9       | miss_case_summary tidiers [0.4 s]
#> 
⠏ |  0       | miss_case_table tidiers
⠋ |  1       | miss_case_table tidiers
⠙ |  2       | miss_case_table tidiers
⠹ |  3       | miss_case_table tidiers
⠸ |  4       | miss_case_table tidiers
⠼ |  5       | miss_case_table tidiers
⠴ |  6       | miss_case_table tidiers
⠦ |  7       | miss_case_table tidiers
⠧ |  8       | miss_case_table tidiers
⠇ |  9       | miss_case_table tidiers
⠏ | 10       | miss_case_table tidiers
✔ | 10       | miss_case_table tidiers [0.3 s]
#> 
⠏ |  0       | miss_prop_summary
⠋ |  1       | miss_prop_summary
⠙ |  2       | miss_prop_summary
⠹ |  3       | miss_prop_summary
⠸ |  4       | miss_prop_summary
⠼ |  5       | miss_prop_summary
⠴ |  6       | miss_prop_summary
⠦ |  7       | miss_prop_summary
⠧ |  8       | miss_prop_summary
⠇ |  9       | miss_prop_summary
⠏ | 10       | miss_prop_summary
⠋ | 11       | miss_prop_summary
⠙ | 12       | miss_prop_summary
⠹ | 13       | miss_prop_summary
✔ | 13       | miss_prop_summary [0.4 s]
#> 
⠏ |  0       | miss_summary
⠋ |  1       | miss_summary
⠙ |  2       | miss_summary
⠹ |  3       | miss_summary
⠸ |  4       | miss_summary
⠼ |  5       | miss_summary
⠴ |  6       | miss_summary
⠦ |  7       | miss_summary
⠧ |  8       | miss_summary
✔ |  8       | miss_summary [0.2 s]
#> 
⠏ |  0       | miss_var_summary tidiers
⠋ |  1       | miss_var_summary tidiers
⠙ |  2       | miss_var_summary tidiers
⠹ |  3       | miss_var_summary tidiers
⠸ |  4       | miss_var_summary tidiers
⠼ |  5       | miss_var_summary tidiers
⠴ |  6       | miss_var_summary tidiers
⠦ |  7       | miss_var_summary tidiers
⠧ |  8       | miss_var_summary tidiers
⠇ |  9       | miss_var_summary tidiers
⠏ | 10       | miss_var_summary tidiers
✔ | 10       | miss_var_summary tidiers [0.3 s]
#> 
⠏ |  0       | miss_var_table tidiers
⠋ |  1       | miss_var_table tidiers
⠙ |  2       | miss_var_table tidiers
⠹ |  3       | miss_var_table tidiers
⠸ |  4       | miss_var_table tidiers
⠼ |  5       | miss_var_table tidiers
⠴ |  6       | miss_var_table tidiers
⠦ |  7       | miss_var_table tidiers
⠧ |  8       | miss_var_table tidiers
⠇ |  9       | miss_var_table tidiers
⠏ | 10       | miss_var_table tidiers
✔ | 10       | miss_var_table tidiers [0.4 s]
#> 
⠏ |  0       | miss_*_pct tidiers
⠋ |  1       | miss_*_pct tidiers
⠙ |  2       | miss_*_pct tidiers
⠹ |  3       | miss_*_pct tidiers
⠸ |  4       | miss_*_pct tidiers
⠼ |  5       | miss_*_pct tidiers
⠴ |  6       | miss_*_pct tidiers
⠦ |  7       | miss_*_pct tidiers
⠧ |  8       | miss_*_pct tidiers
⠇ |  9       | miss_*_pct tidiers
⠏ | 10       | miss_*_pct tidiers
⠋ | 11       | miss_*_pct tidiers
⠙ | 12       | miss_*_pct tidiers
⠹ | 13       | miss_*_pct tidiers
⠸ | 14       | miss_*_pct tidiers
⠼ | 15       | miss_*_pct tidiers
✔ | 15       | miss_*_pct tidiers
#> 
⠏ |  0       | n_complete
⠋ |  1       | n_complete
⠙ |  2       | n_complete
⠹ |  3       | n_complete
⠸ |  4       | n_complete
⠼ |  5       | n_complete
⠴ |  6       | n_complete
⠦ |  7       | n_complete
✔ |  7       | n_complete
#> 
⠏ |  0       | n_miss
⠋ |  1       | n_miss
⠙ |  2       | n_miss
⠹ |  3       | n_miss
⠸ |  4       | n_miss
⠼ |  5       | n_miss
⠴ |  6       | n_miss
⠦ |  7       | n_miss
✔ |  7       | n_miss
#> 
⠏ |  0       | naniar-plots
#> 
⠋ |  1       | naniar-plots
#> 
⠙ |  2       | naniar-plots
#> 
⠹ |  3       | naniar-plots
#> 
⠸ |  4       | naniar-plots
#> 
⠼ |  5       | naniar-plots
#> 
⠴ |  6       | naniar-plots
✔ |  6       | naniar-plots [2.8 s]
#> 
⠏ |  0       | replace_with_na scoped variants
⠋ |  1       | replace_with_na scoped variants
⠙ |  2       | replace_with_na scoped variants
⠹ |  3       | replace_with_na scoped variants
⠸ |  4       | replace_with_na scoped variants
⠼ |  5       | replace_with_na scoped variants
⠴ |  6       | replace_with_na scoped variants
⠦ |  7       | replace_with_na scoped variants
⠧ |  8       | replace_with_na scoped variants
⠇ |  9       | replace_with_na scoped variants
⠏ | 10       | replace_with_na scoped variants
⠋ | 11       | replace_with_na scoped variants
✔ | 11       | replace_with_na scoped variants
#> 
⠙ | 12       | replace_with_na scoped variants
⠹ | 13       | replace_with_na scoped variants
⠸ | 13 1     | replace_with_na scoped variants
⠼ | 14 1     | replace_with_na scoped variants
⠴ | 15 1     | replace_with_na scoped variants
⠦ | 15 2     | replace_with_na scoped variants
⠧ | 16 2     | replace_with_na scoped variants
⠇ | 16 3     | replace_with_na scoped variants
⠏ | 16 4     | replace_with_na scoped variants
⠋ | 17 4     | replace_with_na scoped variants
⠙ | 17 5     | replace_with_na scoped variants
⠏ |  0       | replace_to_na deprecation test
⠋ |  1       | replace_to_na deprecation test
✔ |  1       | replace_to_na deprecation test
#> 
⠏ |  0       | replace_with_na
⠋ |  1       | replace_with_na
⠙ |  2       | replace_with_na
⠹ |  3       | replace_with_na
⠸ |  4       | replace_with_na
⠼ |  5       | replace_with_na
⠴ |  6       | replace_with_na
⠦ |  7       | replace_with_na
⠧ |  8       | replace_with_na
✔ |  8       | replace_with_na
#> 
⠏ |  0       | shadow shift
⠋ |  1       | shadow shift
⠙ |  2       | shadow shift
⠹ |  3       | shadow shift
⠸ |  4       | shadow shift
⠼ |  5       | shadow shift
⠴ |  6       | shadow shift
⠦ |  7       | shadow shift
⠧ |  8       | shadow shift
⠇ |  9       | shadow shift
✔ |  9       | shadow shift
#> 
#> ══ Results ════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
#> Duration: 6.6 s
#> 
#> OK:       231
#> Failed:   5
#> Warnings: 0
#> Skipped:  0
#> 
#> No-one gets it right on their first try
Session info 
devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.4.3 (2017-11-30)
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_AU.UTF-8                 
#>  tz       Australia/Melbourne         
#>  date     2018-01-17
#> Packages -----------------------------------------------------------------
#>  package           * version    date      
#>  assertthat          0.2.0      2017-04-11
#>  backports           1.1.2      2017-12-13
#>  base              * 3.4.3      2017-12-07
#>  bindr               0.1        2016-11-13
#>  bindrcpp          * 0.2        2017-06-17
#>  cli                 1.0.0      2017-11-05
#>  colorspace          1.3-2      2016-12-14
#>  commonmark          1.4        2017-09-01
#>  compiler            3.4.3      2017-12-07
#>  crayon              1.3.4      2017-10-09
#>  datasets          * 3.4.3      2017-12-07
#>  desc                1.1.1      2017-08-03
#>  devtools            1.13.4     2017-11-09
#>  digest              0.6.14     2018-01-14
#>  dplyr               0.7.4      2017-09-28
#>  evaluate            0.10.1     2017-06-24
#>  fontBitstreamVera   0.1.1      2017-02-01
#>  fontLiberation      0.1.0      2016-10-15
#>  fontquiver          0.2.1      2017-02-01
#>  forcats             0.2.0      2017-01-23
#>  formatR             1.5        2017-04-25
#>  gdtools           * 0.1.6      2017-09-01
#>  ggplot2           * 2.2.1.9000 2017-12-06
#>  glue                1.2.0      2017-10-29
#>  graphics          * 3.4.3      2017-12-07
#>  grDevices         * 3.4.3      2017-12-07
#>  grid                3.4.3      2017-12-07
#>  gridExtra           2.3        2017-09-09
#>  gtable              0.2.0      2016-02-26
#>  htmltools           0.3.6      2017-04-28
#>  knitr               1.17       2017-08-10
#>  labeling            0.3        2014-08-23
#>  lazyeval            0.2.1      2017-10-29
#>  magrittr            1.5        2014-11-22
#>  memoise             1.1.0      2017-04-21
#>  methods           * 3.4.3      2017-12-07
#>  munsell             0.4.3      2016-02-13
#>  naniar            * 0.1.4.9100 <NA>      
#>  pillar              1.1.0      2018-01-14
#>  pkgconfig           2.0.1      2017-03-21
#>  plyr                1.8.4      2016-06-08
#>  purrr               0.2.4      2017-10-18
#>  purrrlyr            0.0.2      2017-05-13
#>  R6                  2.2.2      2017-06-17
#>  Rcpp                0.12.14    2017-11-23
#>  rlang               0.1.6      2017-12-21
#>  rmarkdown           1.8        2017-11-17
#>  roxygen2            6.0.1.9000 2017-05-10
#>  rprojroot           1.3-1      2017-12-18
#>  scales              0.5.0.9000 2017-12-06
#>  stats             * 3.4.3      2017-12-07
#>  stringi             1.1.6      2017-11-17
#>  stringr             1.2.0      2017-02-18
#>  svglite             1.2.1      2017-09-11
#>  testthat          * 2.0.0      2017-12-13
#>  tibble              1.4.1      2017-12-25
#>  tidyr               0.7.2      2017-10-16
#>  tidyselect          0.2.3      2017-11-06
#>  tools               3.4.3      2017-12-07
#>  utils             * 3.4.3      2017-12-07
#>  vdiffr              0.2.1      2017-10-09
#>  viridis             0.4.0      2017-03-27
#>  viridisLite         0.2.0      2017-03-24
#>  visdat              0.2.2      2017-12-14
#>  withr               2.1.1      2017-12-19
#>  xml2                1.1.1      2017-01-24
#>  yaml                2.1.16     2017-12-12
#>  source                             
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.3)                     
#>  local                              
#>  cran (@0.1)                        
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.1)                     
#>  local                              
#>  Github (gaborcsardi/crayon@b5221ab)
#>  local                              
#>  CRAN (R 3.4.1)                     
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.2)                     
#>  cran (@0.10.1)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.1)                     
#>  Github (tidyverse/ggplot2@7b5c185) 
#>  cran (@1.2.0)                      
#>  local                              
#>  local                              
#>  local                              
#>  CRAN (R 3.4.1)                     
#>  CRAN (R 3.4.0)                     
#>  cran (@0.3.6)                      
#>  cran (@1.17)                       
#>  CRAN (R 3.4.0)                     
#>  cran (@0.2.1)                      
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.0)                     
#>  local                              
#>  CRAN (R 3.4.0)                     
#>  local                              
#>  CRAN (R 3.4.3)                     
#>  cran (@2.0.1)                      
#>  CRAN (R 3.4.0)                     
#>  cran (@0.2.4)                      
#>  CRAN (R 3.4.0)                     
#>  cran (@2.2.2)                      
#>  cran (@0.12.14)                    
#>  cran (@0.1.6)                      
#>  CRAN (R 3.4.2)                     
#>  Github (klutometis/roxygen@14eae41)
#>  CRAN (R 3.4.3)                     
#>  Github (hadley/scales@d767915)     
#>  local                              
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.1)                     
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.2)                     
#>  local                              
#>  local                              
#>  CRAN (R 3.4.2)                     
#>  CRAN (R 3.4.0)                     
#>  cran (@0.2.0)                      
#>  local                              
#>  CRAN (R 3.4.3)                     
#>  CRAN (R 3.4.0)                     
#>  CRAN (R 3.4.3)
@hadley

This comment has been minimized.

Sign in to view
Member

hadley commented Jan 24, 2018

I wonder if it's related to having two contexts in https://github.com/njtierney/naniar/blob/master/tests/testthat/test-replace-with-na.R. It's possible there's a bug related to multiple contexts in one file.
@hadley

This comment has been minimized.

Sign in to view
Member

hadley commented Jan 24, 2018

Do you still have the problem with a different reporter?
@njtierney

This comment has been minimized.

Sign in to view

njtierney commented Jan 24, 2018

Ooop - sorry, I thought I wrote more about this - I didn't specify context() in the file, this was why it didn't list them in the new reporter, although it did list them in the reporter that runs when I do devtools::check().

@jimhester jimhester added the bug label Mar 29, 2018

@jimhester

This comment has been minimized.

Sign in to view
Member

jimhester commented Mar 29, 2018

marking this as a bug, as we should still be reporting test failures even if there is no context (by having a default context).

jimhester added a commit to jimhester/testthat that referenced this issue Apr 2, 2018

@jimhester
Generate a context if one does not exist 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter. This generates a
context from the filename if there is not already a defined context,
fixing the issue.

Fixes r-lib#700
Fixes r-lib#705
dff9aea

@jimhester jimhester referenced this issue Apr 2, 2018

Merged

Generate a context if one does not exist #738

jimhester added a commit to jimhester/testthat that referenced this issue Apr 2, 2018

@jimhester
Generate a context if one does not exist 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter. This generates a
context from the filename if there is not already a defined context,
fixing the issue.

Fixes r-lib#700
Fixes r-lib#705
3dcc732

jimhester added a commit to jimhester/testthat that referenced this issue Apr 2, 2018

@jimhester
Generate a context if one does not exist 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter and a different (unreported) error in
the JUnit reporter. This generates a context from the filename if there
is not already a defined context, fixing the issue.

Fixes r-lib#700
Fixes r-lib#705
b85e069

@jimhester jimhester closed this in #738 Apr 4, 2018

jimhester added a commit that referenced this issue Apr 4, 2018

@jimhester
Generate a context if one does not exist (#738) 
Test files without an explicit `context()` would cause

    Error in x[[method]](...) : attempt to apply non-function

errors in the Progress reporter and a different (unreported) error in
the JUnit reporter. This generates a context from the filename if there
is not already a defined context, fixing the issue.

Fixes #700
Fixes #705
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
42e3a79
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment