Re-consider where helper files are sourced in load_all()

[jimhester]

Issue by berndbischl
Thursday Apr 14, 2016 at 15:13 GMT
Originally opened as r-lib/devtools#1146

---

Your NEWS state this

load_all() now sources all test helpers if you use testthat. This makes it much easier to interactively run tests (#1125). load_all() also correctly handles unix and windows subdirectories within R (@gaborcsardi, #1102)

In my setup I have a helper.R file, in tests/testthat which creates a few (data) objects I want to use in my unit tests (among other stuff).

You can reproduce the problem by creating a basically empty (but valid) package, then adding such a file under tests/testthat

data(Sonar, package = "mlbench")
x = Sonar

This now breaks, after the new devtools update

> load_all()
Loading minipkg
Error in eval(expr, envir, enclos) : object 'Sonar' not found

mlbench would usually be in my SUGGESTS (and I want it to be there) but even adding it to DEPENDS and adding and explicit call to library("mlbench") in the helper.R does not remove the problem.

This now breaks our complete devel workflow.

Any advice?

[jimhester]

Comment by jimhester
Thursday Apr 14, 2016 at 17:29 GMT

---

The problem is the helpers are sourced in the package environment, but data() by default loads the objects in the global environment. Explicitly loading into the calling environment fixes the issue. (see mlr-org/mlr#835)

@hadley maybe we should add a data() shim which makes this the default?

[jimhester]

Comment by berndbischl
Thursday Apr 14, 2016 at 19:23 GMT

---

Explicitly loading into the calling environment fixes the issue.

Cool that you are even fixing our package for us :) Thx a lot!

@hadley maybe we should add a data() shim which makes this the default?

I would seriously suggest that. What happened on our side was that I wanted to show a student assistant how the mlr unit tests work. Apparently at that point in time we got the new devtools version on her laptop. After calling load_all on the package, and seeing the error, we had 3 technically competent people sitting around the laptop for 30 min thinking "wtf is happening".
The error is extremely "unintuitive" and very hard to debug. I basically had to guess what happened, and downgrade devtools to figure this out.

[jimhester]

Comment by imanuelcostigan
Saturday Apr 16, 2016 at 00:22 GMT

---

I have a package that creates some option settings through the .onAttach hook. My test helper files require these options. Unfortunately, devtools runs the package hooks after sourcing the test helpers and breaks things for me. Why can't the test helper files be loaded after the package is attached by load_all()?

[jimhester]

Comment by imanuelcostigan
Sunday May 01, 2016 at 09:45 GMT

---

@hadley would you be open to me submitting a PR with the effect of sourcing the test helper files after running hooks?

PS - check() passes without error after making this change.

[jimhester]

Comment by hadley
Sunday May 01, 2016 at 10:02 GMT

---

Sure

[jimhester]

Comment by berndbischl
Sunday May 01, 2016 at 10:28 GMT

---

one other question from my side, that i just noticed.

in the past, i also created a file called helper_zzz.R
this eg set a seed for my unit tests. and this seed should ONLY be set for the unit tests.
now it is always used (when i call load_all on the package)

so i guess my question is:
how can i do "state changes" in my helper files, which are only executed when i am in "testthat unit testing mode" ?

[jimhester]

Comment by hadley
Friday Jun 03, 2016 at 17:09 GMT

---

@berndbischl maybe condition on interactive()?

[jimhester]

Comment by jennybc
Friday Jun 03, 2016 at 18:27 GMT

---

It feels like part of the pain around test helpers is that they are used for setup but there's no similarly official way to do corresponding tear down. I have a faux test file named test-zz-clean-up.R that does this. Is that awful?

[jimhester]

Comment by hadley
Friday Jun 03, 2016 at 18:29 GMT

---

What are you typically tearing down?

[jimhester]

Comment by jennybc
Friday Jun 03, 2016 at 18:32 GMT

---

OK this will be very specific. But I remove an OAuth token from an environment internal to the package. I delete some local files. I delete some things from Google Drive.

[jimhester]

Comment by hadley
Friday Jun 03, 2016 at 18:33 GMT

---

Maybe what we need for the non-idempotent stuff is to simply source a setup.R before running the tests and a teardown.R after the tests?

[jimhester]

Comment by jennybc
Friday Jun 03, 2016 at 18:36 GMT

---

Is there something like on.exit() in this context?

[jimhester]

Comment by hadley
Friday Jun 03, 2016 at 18:41 GMT

---

No, but that would be another approach - if you wanted to keep setup and tear down in the same file together. Could have something that allowed you to register setup and teardown code for each run.

I'm thinking something like:

register_test_something(
  setup = file.create("x"),
  teardown = unlink("x")
)

[jimhester]

Comment by berndbischl
Saturday Jun 04, 2016 at 09:30 GMT

---

@berndbischl maybe condition on interactive()?

we also run our unit tests in non-interactive mode via shell scripts / from the console. actually we do that most often.

[jimhester]

Comment by pitakakariki
Tuesday Jun 07, 2016 at 04:09 GMT

---

@berndbischl maybe condition on interactive()?

I think interactive() returns TRUE whether you've called load_all() or test().

[jimhester]

Closed by #41