Impossible to save a Dataset when using torch/image

[Cadene]

Please try the following code.

local tnt = require 'torchnet'
local image = require 'image'

local mode = 'train'
local mnist = require 'mnist'
local dataset = mnist[mode .. 'dataset']()

local listdataset = tnt.ListDataset{ 
   list = torch.range(1, dataset.data:size(1)):long(),
   load = function(idx)
      return {
         input  = image.scale(dataset.data[idx],10,10),
         target = torch.LongTensor{dataset.label[idx] + 1},
      } 
   end,
}

torch.save('image.scale', image.scale)     -- works
torch.save('listdataset.tnt', listdataset) -- doesnt work
torch.save('image',image)                  -- doesnt work

/home/cadene/torch/install/bin/luajit: /home/cadene/torch/install/share/lua/5.1/torch/File.lua:141: Unwritable object <function> at <?>.tnt.ListDataset.load.image.float.scaleBicubic
stack traceback:
    [C]: in function 'error'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:141: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:235: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:235: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:235: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:235: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:200: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:235: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:220: in function 'writeObject'
    /home/cadene/torch/install/share/lua/5.1/torch/File.lua:388: in function 'save'
    src/bugserialize.lua:18: in main chunk
    [C]: in function 'dofile'
    ...dene/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
    [C]: at 0x00405ce0

[lvdmaaten]

This is expected behavior. Your dataset contains image as an upvalue, which means you're trying to serialize the full image package. The image package contains C-code that cannot be serialized. Hence the error.

Try moving the require 'image' into the load closure.

[Cadene]

Ok I can save my dataset in the main thread, then load it in closure and add some transformations.
Thanks :)

[Cadene]

Removing local image solved the issue. Because image isn't anymore an upvalue.

local tnt = require 'torchnet'
require 'image'

local mode = 'train'
local mnist = require 'mnist'
local dataset = mnist[mode .. 'dataset']()

local listdataset = tnt.ListDataset{ 
   list = torch.range(1, dataset.data:size(1)):long(),
   load = function(idx)
      return {
         input  = image.scale(dataset.data[idx],10,10),
         target = torch.LongTensor{dataset.label[idx] + 1},
      } 
   end,
}

torch.save('image.scale', image.scale)     -- works
torch.save('listdataset.tnt', listdataset) --  works

or

local tnt = require 'torchnet'
local image = require 'image'

local scale = image.scale -- an upvalue -> serialized
-- scale = image.scale    -- not an upvalue

local mode = 'train'
local mnist = require 'mnist'
local dataset = mnist[mode .. 'dataset']()

local listdataset = tnt.ListDataset{ 
   list = torch.range(1, dataset.data:size(1)):long(),
   load = function(idx)
      return {
         input  = scale(dataset.data[idx],10,10),
         target = torch.LongTensor{dataset.label[idx] + 1},
      } 
   end,
}

torch.save('image.scale', image.scale)     -- works
torch.save('listdataset.tnt', listdataset) --  works