reach out for help

[jlxAtNovozymes]

Hi, Keydana. first of all, thanks a lot for wrting such an amazing book for deep learning. It's my first book to deep learning and I still read it. but when I went to luz chapter, I had a question in my mind: how to use L-BFGS optimizer with luz, could you give me a simple example about it, thanks a lot

[skeydan]

Hi, happy to hear you enjoy it!

That's a good question! Basically the way to do it is write a custom loop, similar to the examples given here: https://mlverse.github.io/luz/articles/custom-loop.html:

library(luz)
library(torch)

net <- nn_module(
  "Net",
  initialize = function(num_hidden) {
    self$fc1 <- nn_linear(1, num_hidden)
    self$fc2 <- nn_linear(num_hidden, 1)
  },
  forward = function(x) {
    x <- self$fc1(x)
    x <- nnf_relu(x)
    x <- self$fc2(x)
    x
  },
  step = function() {
    if (ctx$training) {
      closure <- function() {
        pred <- ctx$model(ctx$input)
        loss <- ctx$loss_fn(pred, ctx$target)
        loss$backward()
        loss
      }
      ctx$loss <- ctx$opt$step(closure)
    } else {
      pred <- ctx$model(ctx$input)
      ctx$loss <- ctx$loss_fn(pred, ctx$target)
    }
  }
)

train_data <- list(torch_arange(1, 100)$unsqueeze(2), 2 * torch_arange(1, 100)$unsqueeze(2))

fitted <- net %>% 
  setup(
    loss = nn_mse_loss(),
    optimizer = optim_lbfgs
  ) %>% 
  set_hparams(num_hidden = 16) %>%
  set_opt_hparams(line_search_fn = "strong_wolfe") %>%
  fit(train_data, epochs = 3, valid_data = train_data)

[jlxAtNovozymes]

Hi, Keydana. A really big thanks. BTW you wrote a really awesome book, I'm new to deep learning, I think your books make me easy to understand logics behind the scene. Really really fantastic work!

[skeydan]

Thank you!!