Checkpointer cannot restore when using fields with OffsetArrays.

[ali-ramadhan]

For now the 10 tests affected by this are skipped with @test_skip and will show up as "Broken".

Maybe we just have to define convert for Fields with OffsetArrays?

    Testing checkpointing...
WARNING: Forcing functions will not be serialized!
[Checkpointer] Serializing model to disk: ./test_model_checkpoint_0.jld
[Checkpointer] Reconstructing FFT plans...
WARNING: Forcing functions will not be serialized!
[Checkpointer] Serializing model to disk: ./test_model_checkpoint_5.jld
[Checkpointer] Reconstructing FFT plans...
Deserializing model from disk: test_model_checkpoint_5.jld
┌ Warning: type Oceananigans.FaceFieldX{OffsetArrays.OffsetArray{Core.Float64,3,Core.Array{Core.Float64,3}},Oceananigans.RegularCartesianGrid{Core.Float64,Base.StepRangeLen{Core.Float64,Base.TwicePrecision{Core.Float64},Base.TwicePrecision{Core.Float64}}}} not present in workspace; reconstructing
└ @ JLD ~/.julia/packages/JLD/1BoSz/src/jld_types.jl:703
┌ Warning: type OffsetArrays.OffsetArray{Core.Float64,3,Core.Array{Core.Float64,3}} not present in workspace; reconstructing
└ @ JLD ~/.julia/packages/JLD/1BoSz/src/jld_types.jl:703
Checkpointing: Error During Test at /home/alir/Oceananigans.jl/test/runtests.jl:293
  Got exception outside of a @test
  MethodError: Cannot `convert` an object of type getfield(JLD, Symbol("##Oceananigans.FaceFieldX{OffsetArrays.OffsetArray{Core.Float64,3,Core.Array{Core.Float64,3}},Oceananigans.RegularCartesianGrid{Core.Float64,Base.StepRangeLen{Core.Float64,Base.TwicePrecision{Core.Float64},Base.TwicePrecision{Core.Float64}}}}#384")) to an object of type FaceFieldX
  Closest candidates are:
    convert(::Type{T}, !Matched::T) where T at essentials.jl:154
  Stacktrace:
   [1] jlconvert(::Type{VelocityFields}, ::JLD.JldFile, ::Ptr{UInt8}) at /home/alir/.julia/packages/JLD/1BoSz/src/jld_types.jl:434
   [2] read_scalar(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type) at /home/alir/.julia/packages/JLD/1BoSz/src/JLD.jl:398
   [3] read(::JLD.JldDataset) at /home/alir/.julia/packages/JLD/1BoSz/src/JLD.jl:370
   [4] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /home/alir/.julia/packages/JLD/1BoSz/src/JLD.jl:502
   [5] jlconvert(::Type{Model{CPU}}, ::JLD.JldFile, ::Ptr{UInt8}) at /home/alir/.julia/packages/JLD/1BoSz/src/jld_types.jl:387
   [6] read_scalar(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type) at /home/alir/.julia/packages/JLD/1BoSz/src/JLD.jl:398
   [7] read(::JLD.JldDataset) at /home/alir/.julia/packages/JLD/1BoSz/src/JLD.jl:370
   [8] read(::JLD.JldFile, ::String) at /home/alir/.julia/packages/JLD/1BoSz/src/JLD.jl:346
   [9] restore_from_checkpoint(::String) at /home/alir/Oceananigans.jl/src/output_writers.jl:94
   [10] run_thermal_bubble_checkpointer_tests() at /home/alir/Oceananigans.jl/test/test_output_writers.jl:34
   [11] top-level scope at /home/alir/Oceananigans.jl/test/runtests.jl:295
   [12] top-level scope at /build/julia/src/julia-1.1.0/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
   [13] top-level scope at /home/alir/Oceananigans.jl/test/runtests.jl:294
   [14] top-level scope at /build/julia/src/julia-1.1.0/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
   [15] top-level scope at /home/alir/Oceananigans.jl/test/runtests.jl:290
   [16] top-level scope at /build/julia/src/julia-1.1.0/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
   [17] top-level scope at /home/alir/Oceananigans.jl/test/runtests.jl:14
   [18] include(::String) at ./client.jl:403
   [19] top-level scope at none:0

[ali-ramadhan]

The way forward here might be to just defined a CheckpointedModel struct with stuff we want to save to the checkpoint file. That way we can keep the struct simple and we won't have to worry about all sorts of weird things being serialized to disk.