Forcing dot-exports and/or renaming exports

[mmuurr]

While the dot-prefix has obvious advantages for implied convention behavior, it'd be useful at times to be able to force exports of an object that has a dot prefix, e.g. in a file module.R:

.f <- function() NULL
modules::export(.f)

Then at the R prompt:

> foo <- modules::use("module.R")

 Error in exportExtract2List(module) : exports not defined: .f

(I realize we could add an additional line in the module file (f <- .f) to overcome this, but I'm wondering if we can explicitly specify what should/shouldn't be exported regardless of the naming convention.)

Similarly, it doesn't appear that we can rename objects when explicitly exporting them, e.g., this might be nice in module.R:

.f <- function() NULL
modules::export(f = .f)

Then at the R prompt:

> foo <- modules::use("module.R")
> foo$f()

NULL

[wahani]

it'd be useful at times to be able to force exports of an object that has a dot prefix,

Well, it is of course possible. In R and other systems (e.g. file systems and such) the .function convention is generally accepted as hide this element. The dot is of course not important, but to have a convention you can rely on is the thing I would like to maintain. But maybe you can describe in what situations this would be useful, maybe I am missing the point.

If you need access to private functions, e.g. for testing, you can allway use modules::extend(foo, {.f()}).

Similarly, it doesn't appear that we can rename objects when explicitly exporting them, e.g., this might be nice in module.R:

You are right. That is not possible. This is also the explanation for #20 by the way:

modules::export is not managing a list objects. Instead it only stores strings/characters in a hidden object. So we declare names of objects, not the objects themselves. So when you call:

modules::export(f)

we deparse the expression f, such that it is equivalent to:

modules::export("f")

It is one of those times where non-standard evaluation is probably misleading. I will stop here and make a suggestion in #20. I think when we allow for something like exportObject(<name> = <object>) we implicitly allow to export dot functions.

[wahani]

Got confused with #19 and #20. So here it comes: We may add to the functionality of export or introduce a new function like exportObject[s]. This would then allow for things like:

m <- modules::module({
  modules::export(bar = foo) # option a
  modules::exportObject(bar = foo) # option b
  foo <- function() "foo"
})

The problem with this is that the export declaration can be placed before the function definitions. The reason for this is that in longer modules it is often useful to place imports and exports into a header block like:

m <- modules::module({
  # imports
  modules::import(...)
  ...
  # exports
  modules::export(foo, bar, etc)
  # implementations
  foo <- function() "foo"
  ...
})

So we would need to evaluate everything lazily. That means expressions like modules::export(f1 = my_exports$f1) can be placed before my_exports is defined. Although this sounds confusing, we should still allow to place exports at the top, basically organizing the namespace if you will, and that is what it takes.

Allowing to rename while exporting will also allow to export dot functions.

Introducing this feature will require a rewrite of export. This requires some more work but certainly feels much better than having an additional function like exportObject. Also the semantics should allow for:

modules::export(
  foo = function(a = 1) a
)

Which should definitely be avoided, but is possible to write and should work.

@mmuurr maybe you can give some feedback if this would be an improvement for your work and if the examples make sense. While renaming in export seems like a nice feature, this will need a couple of hours with the potential for more, because of all the implications with non-standard evaluation these things get easily out of hand.

Currently the simplest way of doing this:

m <- modules::module({
  modules::export(bar = foo)
  foo <- function() "foo"
})

is

m <- modules::module({
  modules::export(bar)
  bar <- foo <- function() "foo"
})

But why would we do this? We already have a limited scope and can choose the appropriate names right away. Even more, it can be very confusing when we first have foo, but export bar. Instead of properly renaming all references to foo while refactoring we may encourage to just rename the export. It feels like this feature may be more useful for imports? Here are the two reasons I can think of to have such a feature:

• to circumvent the dot function convention (lets discuss this one separately, because there are other much simpler ways of getting there)
• to directly export elements from a sub module (see examples below)
• @mmuurr maybe you can add if I am missing something

For the sake of completeness, elements from a sub-module can currently be exported when we write:

m <- modules::module({
  .sm <- modules::module({
    foo <- function() "foo"
  })
  bar <- function() "bar"
  foo <- .sm$foo
})

or

m <- modules::module({
  .sm <- modules::module({
    foo <- function() "foo"
  })
  modules::expose(.sm) # just reexport from .sm
  bar <- function() "bar"
})

and some other ways.

[mmuurr]

I hadn't realized at first that the exporting mechanism was just storing strings and using non-standard evaluation (NSE) ... it makes sense, though, in the 'declare an export before defining the exported object itself' pattern. In my patterns, I tend to place all the exports at the end of a code block, akin to a 'return value' in spirit.

And stepping-back for a moment, a use case (in fact, the use case that I encountered) for allowing explicitly renamed exports is when one is converting an existing script to a modules-compliant module. This is especially true when one might have multiple scripts that are to be exposed via a common 'interface' (e.g. two different implementations of some modeling procedure, to be contrasted and compared). Those scripts may be long, relatively complex, and even authored by different individuals. And rather than worry about renaming elements in the script or adjusting much code, all that's needed to port them to a modules format is replacing any library(...) (and/or requires(...)) statements with the appropriate modules::import(...) call, then deciding which end-of-script objects are to be exported. This allows one to leave everything else in the script intact (in fact, one could at that point just source(...) most of the original script) and the last few lines of code would simply be shimming the script objects to the appropriate to-be-exported objects.

That shimming could be as simple as declaring some 'aliases' via assignment then exporting:

foo <- some_script_object_1
bar <- some_script_object_2
baz <- .some_script_object_3_with_a_preceding_dot
modules::export(foo, bar, baz)

... and this is basically exactly what I'm doing now. But this is where it just started to feel a bit clunky and I thought it'd be easy-enough to combine the renaming into an explicit export call with named parameters. The NSE obviously makes this a bit trickier and I hadn't poked hard enough at the modules package yet to see why (though it makes sense now). The old-school tidyverse way of providing escape-hatches (e.g. export_) is nice compromise (though they're moving away from that mechanism), but yeah, generally mixing NSE and standard evaluation into the same calls is tough. In the example above, rather than create some 'new' (though lightweight, thanks to copy-on-write semantics) objects, I think it'd be nice to simply have something like:

modules::export(
  foo = some_script_object_1
 ,bar = some_script_object_2
 ,baz = .some_script_object_3_with_a_preceding_dot
 ,.and_maybe_even_a_preceded_dot_additional_object = some_other_script_object)

, which then opens the door for do.call(...) wrapping, which was my first naïve attempt at this pattern :-)

Of course in all of this I realize there are tons of coding patterns in the wild that people have adopted and that designing generic solutions that accommodate all of them is nearly impossible :-). So, regardless of what might come out of this thread, thanks for the modules package and all your hard work!

[wahani]

It is quite interesting that most users of the modules package (those who report on GitHub anyways) do the same thing: working with other peoples code and trying to make it work. It is kind of related to what I had in mind when I wrote the package, but just a different use case. In that scenario I can understand why renaming may be a thing you want to do.

While reviewing the code I realized that a large portion of what we talked about here can be implemented easily. You can try and install the draft with:

remotes::install_github("wahani/modules", "rename-exports")

Most parts are working, but the do.call part fails. From a users perspective the behavior must feel completely erratic. Some examples to illustrate the problem:

m <- modules::module({
  sm <- module({
    dep <- 1
    fun <- function() dep
  })
  foo <- function() "foo"
  .foo <- foo
  do.call(export, list("foo", .foo = .foo, bar = sm$fun))
  export(bar1 = sm$fun)
})

m$foo()
m$.foo()
m$bar1()
m$bar() # errors

This is the part where NSE will get out of hands. The reason we see this error is, that the do.call expression translates into:

## do.call(export, list("foo", .foo = .foo, bar = sm$fun))
export("foo", .foo = function() "foo", bar = function() dep)

It means that the exported objects are NOT the objects you pass as arguments to do.call. Instead their function bodies will get reevaluated in the environment where export is called and they loose their context. This will lead to some extremely hard to find bugs. Actually this is so tricky to find out, that I am thinking about throwing an error as soon as you try to use do.call and export together.

Feedback is of course welcome.

[wahani]

The only save way of using do.call would be:

do.call(export, list("foo", .foo = ".foo", bar = "sm$fun"))

Depending on how one thinks about this function, this may only add to the confusion and add little benefit.

[wahani]

@mmuurr any feedback on this?

[mmuurr]

@wahani sorry for the laggy reply ... yes, the renaming feature is quite nice, I've just been experimenting with it. I think it accomplishes most of the underlying goal of carefully managing how to export other 'source()-d' objects.

I also noticed that in:

mod <- use("file.R")

mod is a list, and additional list mutation functions (e.g. those in the purrr package) seem to work, opening the door for additional renaming after the export (i.e. by the calling code). In my testing, this appears to work just fine, but curious if you anticipate anything breaking because of that.

Either way, thanks much; it's great work.

[wahani]

mod is a list

yes. a module is always represented as list.

In my testing, this appears to work just fine, but curious if you anticipate anything breaking because of that.

Nothing I can think of. There are unit-tests covering all known (to me) use-cases and none needed to be changed. So I do not expect this to be a breaking change.

Thanks for reporting back!