Issue: gremlins.js gremlins.species.scroller() crashes Chromote session

[DivadNojnarg]

Hi @schloerke ,

as discussed, I open this issue just for recording it. I was trying to run Monkey testing with shinytest2, leveraging gremlins.js.

Below is my test example:

app.R

ui <- fluidPage(
    sliderInput("obs", "Number of observations:", min = 0, max = 1000, value = 500),
    plotOutput("distPlot")
  )

  server <- function(input, output, session) {
    output$distPlot <- renderPlot({
      hist(rnorm(input$obs))
    })
  }
  shinyApp(ui, server)

shinytest2 code:

plop <- shinytest2::AppDriver$new(
  path = "app.R",
  name = "foo"
)

Setting the gremlins.js require to inject a JS script in the DOM. Notice, sometimes https://unpkg.com/gremlins.js maybe blocked by corporate networks. A solution is to serve the gremlins JS with the app using local script copy and a call to addResourcePath("gremlins", "inst/js")... :

plop$execute_js("
(function() {
  var s = document.createElement('script');
  s.src = 'https://unpkg.com/gremlins.js'; // or './gremlins/gremlins.js' if issues
  document.body.appendChild(s);
})();")
Sys.sleep(2)

Then just check if the script is in the DOM and if we can get window.gremlins:

plop$get_html("script", outer_html = TRUE)
 [1] "<script type=\"application/shiny-singletons\"></script>"                                                                                                                                                                         
 [2] "<script type=\"application/html-dependencies\">jquery[3.5.1];shiny-css[1.6.0];shiny-javascript[1.6.0];shiny-testmode[1.6.0];ionrangeslider-javascript[2.3.1];strftime[0.9.2];ionrangeslider-css[2.3.1];bootstrap[3.4.1]</script>"
 [3] "<script src=\"shared/jquery.min.js\"></script>"                                                                                                                                                                                  
 [4] "<script src=\"shared/shiny.min.js\"></script>"                                                                                                                                                                                   
 [5] "<script src=\"shared/shiny-testmode.js\"></script>"                                                                                                                                                                              
 [6] "<script src=\"shared/ionrangeslider/js/ion.rangeSlider.min.js\"></script>"                                                                                                                                                       
 [7] "<script src=\"shared/strftime/strftime-min.js\"></script>"                                                                                                                                                                       
 [8] "<script src=\"shared/bootstrap/js/bootstrap.min.js\"></script>"                                                                                                                                                                  
 [9] "<script src=\"shared/bootstrap/accessibility/js/bootstrap-accessibility.min.js\"></script>"                                                                                                                                      
[10] "<script src=\"./gremlins/gremlins.min.js\"></script>" 
plop$execute_js("return typeof window.gremlins")
# [1] "object"

If it is an object this is fine, if not this is likely because the script was blocked (see above).

Next step is to set up the gremlins horde. This is where I get the issue. They provide a plug and play script:

(function() {
    function callback() {
        gremlins.createHorde({
            species: [gremlins.species.clicker(),gremlins.species.toucher(),gremlins.species.formFiller(),gremlins.species.scroller(),gremlins.species.typer()],
            mogwais: [gremlins.mogwais.alert(),gremlins.mogwais.fps(),gremlins.mogwais.gizmo()],
            strategies: [gremlins.strategies.distribution()]
        }).unleash();
    }
    var s = document.createElement("script");
    s.src = "https://unpkg.com/gremlins.js";
    if (s.addEventListener) {
        s.addEventListener("load", callback, false);
    } else if (s.readyState) {
        s.onreadystatechange = callback;
    }
    document.body.appendChild(s);
    })()

I never managed to get it working on CI/CD. I realised later that the gremlins.species.scroller() is crashing the Chromote session:

plop$get_log()
{shinytest2} R  info  08:40:31.30 Start AppDriver initialization
{shinytest2} R  info  08:40:31.38 Starting Shiny app
{shinytest2} R  info  08:40:33.65 Creating new chromote session
{shinytest2} R  info  08:40:34.94 Navigating to Shiny app
{shinytest2} R  info  08:40:35.06 Injecting shiny-tracer.js
{chromote}   JS info  08:40:35.08 shinytest2; jQuery found
{chromote}   JS info  08:40:35.10 shinytest2; Waiting for shiny session to connect
{chromote}   JS info  08:40:35.10 shinytest2; Loaded
{shinytest2} R  info  08:40:35.10 Waiting for Shiny to become ready
{chromote}   JS info  08:40:35.16 shinytest2; Connected
{chromote}   JS info  08:40:35.21 shinytest2; shiny:busy
{shinytest2} R  info  08:40:35.24 Waiting for Shiny to become idle for 500ms within 1e+05ms
{chromote}   JS info  08:40:35.24 shinytest2; Waiting for Shiny to be stable
{chromote}   JS info  08:40:35.31 shinytest2; shiny:idle
{chromote}   JS info  08:40:35.31 shinytest2; shiny:value distPlot
{chromote}   JS info  08:40:35.81 shinytest2; Shiny has been idle for 500ms
{shinytest2} R  info  08:40:35.81 Shiny app started
{shiny}      R  error ----------- Loading required package: shiny
{shiny}      R  error ----------- Warning in loadSupport(appDir, renv = sharedEnv, globalrenv = NULL) :
{shiny}      R  error -----------   Loading R/ subdirectory for Shiny application, but this directory appears to contain an R package. Sourcing files in R/ may cause unexpected behavior.
{shiny}      R  error ----------- ℹ Loading shinyValidatorTest
{shiny}      R  error ----------- Running application in test mode.
{shiny}      R  error ----------- 
{shiny}      R  error ----------- Listening on http://127.0.0.1:29897
{shiny}      R  error ----------- SEND {"config":{"workerId":"","sessionId":"5e340b9528e4bdc6d564adccfac4480b","user":null}}
{shiny}      R  error ----------- RECV {"method":"init","data":{"obs":500,".clientdata_output_distPlot_width":962,".clientdata_output_distPlot_height":400,".clientdata_output_distPlot_bg":"rgb(255, 255, 255)",".clientdata_output_distPlot_fg":"rgb(51, 51, 51)",".clientdata_output_distPlot_accent":"rgb(51, 122, 183)",".clientdata_output_distPlot_font":{"families":["Helvetica Neue","Helvetica","Arial","sans-serif"],"size":"14px"},".clientdata_output_distPlot_hidden":false,".clientdata_pixelratio":1,".clientdata_url_protocol":"http:",".clientdata_url_hostname":"127.0.0.1",".clientdata_url_port":"29897",".clientdata_url_pathname":"/",".clientdata_url_search":"",".clientdata_url_hash_initial":"",".clientdata_url_hash":"",".clientdata_singletons":""}}
{shiny}      R  error ----------- SEND {"busy":"busy"}
{shiny}      R  error ----------- SEND {"recalculating":{"name":"distPlot","status":"recalculating"}}
{shiny}      R  error ----------- SEND {"recalculating":{"name":"distPlot","status":"recalculated"}}
{shiny}      R  error ----------- SEND {"busy":"idle"}
{shiny}      R  error ----------- SEND {"errors":{},"values":{"distPlot":{"src":"data:image/png;[base64 data]","width":962,"height":400,"alt":"Plot object","coordmap":{"panels":[{"domain":{"left":-3.24,"right":3.24,"bottom":-3.76,"top":97.76},"range":{"left":59.04,"right":931.76,"bottom":325.56,"top":58.04},"log":{"x":null,"y":null},"mapping":{}}],"dims":{"width":962,"height":400}}}},"inputMessages":[]}
{shiny}      R  error ----------- session has stoppedWarning message:
In callback(...) :
  Chromote has received a Inspector.targetCrashed event. This means that the ChromoteSession has probably crashed.

As I said, I am not able to reproduce locally where it works well and with the same Chrome version (Google Chrome 98.0.4758.102).

In practice, I don't think users will ever need the scroller gremlins since this app only requires click and touch. I'll add a warning in the upcoming vignette.

[DivadNojnarg]

Side note.

A working gremlins config for this app. I set up a custom clicker that will only target the slider. I also assume some touch events (mobile, ...), set probability distribution of triggering each species. Also adding some quick log message at the end of the promise:

const customClicker = gremlins.species.clicker({
      // which mouse event types will be triggered
      clickTypes: ['click'],
      // Click only if element has id obs
      canClick: (element) => element.id === 'obs',
      // by default, the clicker gremlin shows its action by a red circle
      // overriding showAction() with an empty function makes the gremlin action invisible
      showAction: (x, y) => {},
    });
    
    gremlins.createHorde({
      randomizer: new gremlins.Chance(1234), // repeatable
      species: [customClicker, gremlins.species.toucher()],
      mogwais: [gremlins.mogwais.alert(),gremlins.mogwais.gizmo()],
      strategies: [
        gremlins.strategies.distribution({
          distribution: [0.8, 0.2], // custom 
          delay: 10 // default
        })
      ]
    }).unleash().then(() => {
      console.log('Gremlins test success')
    });

[schloerke]

@DivadNojnarg

Debug idea...

• After initializing the app, we can set a callback on the ChromoteSession event being thrown and try to see what is going on. I'm not too hopeful it'll provide any useful information, but it's worth a try! If it does provide some sort of hint, I'll add the callback by default.

app <- AppDriver$new(....)

# Set up the crash callback
chromote_session <- app$get_chromote_session()
chromote_session$Inspector$targetCrashed(function(...) {
  # Notify event is running
  message("Running Inspector.targetCrashed callback!")
  # Display args being passed in
  str(list(...))
  # Display current app log
  print(app$get_log())
})

# Perform actions as normal to trigger the crash
....

[DivadNojnarg]

@schloerke: I can give it a try on a test branch.

[DivadNojnarg]

Covered in this vignette.

[schloerke]

Fixed in #140