Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work in progress in migrating the TensorFlow.jl into the Julia 1.0.0 #419

Merged
merged 64 commits into from
Aug 29, 2018

Conversation

adamryczkowski
Copy link
Contributor

Don't merge it blindly, because this patch still contains at least one bug:

julia> import TensorFlow
[ Info: Precompiling TensorFlow [1d978283-2c37-5f34-9a8e-e9c0ece82495]
┌ Warning: Loading a new version of TensorFlow.jl for the first time. This initial load can take around 5 minutes as code is precompiled; subsequent usage will only take a few seconds.
└ @ TensorFlow ~/.julia/packages/TensorFlow/qrmc9/src/TensorFlow.jl:6
ERROR: LoadError: UndefVarError: broadcast_c not defined
Stacktrace:
 [1] include at ./boot.jl:317 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1038
 [3] include(::Module, ::String) at ./sysimg.jl:29
 [4] top-level scope at none:2
 [5] eval at ./boot.jl:319 [inlined]
 [6] eval(::Expr) at ./client.jl:389
 [7] top-level scope at ./none:3
in expression starting at /home/adam/.julia/packages/TakingBroadcastSeriously/WQSQT/src/TakingBroadcastSeriously.jl:3
ERROR: LoadError: LoadError: Failed to precompile TakingBroadcastSeriously [fa778a35-eff5-5ebc-91c5-11ea73160f07] to /home/adam/.julia/compiled/v1.1/TakingBroadcastSeriously/RVPKc.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] _require(::Base.PkgId) at ./logging.jl:311
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] require(::Module, ::Symbol) at ./logging.jl:311
 [7] include at ./boot.jl:317 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1038
 [9] include at ./sysimg.jl:29 [inlined]
 [10] include(::String) at /home/adam/.julia/packages/TensorFlow/qrmc9/src/TensorFlow.jl:2
 [11] top-level scope at none:0
 [12] include at ./boot.jl:317 [inlined]
 [13] include_relative(::Module, ::String) at ./loading.jl:1038
 [14] include(::Module, ::String) at ./sysimg.jl:29
 [15] top-level scope at none:2
 [16] eval at ./boot.jl:319 [inlined]
 [17] eval(::Expr) at ./client.jl:389
 [18] top-level scope at ./none:3
in expression starting at /home/adam/.julia/packages/TensorFlow/qrmc9/src/ops.jl:5
in expression starting at /home/adam/.julia/packages/TensorFlow/qrmc9/src/TensorFlow.jl:189
ERROR: Failed to precompile TensorFlow [1d978283-2c37-5f34-9a8e-e9c0ece82495] to /home/adam/.julia/compiled/v1.1/TensorFlow/IhIhf.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] macro expansion at ./logging.jl:311 [inlined]
 [5] _require(::Base.PkgId) at ./loading.jl:941
 [6] require(::Base.PkgId) at ./loading.jl:852
 [7] require(::Module, ::Symbol) at ./logging.jl:311

I believe that the proper solution is to factor out the TakingBroadcastSeriously dependency, but that task is still to difficult for me to do.

@malmaud
Copy link
Owner

malmaud commented Aug 20, 2018

Thanks a ton for doing this work! I think you're right that we should get the broadcasting stuff using the new Julia 1.0 techniques as a first step. I'm cautiously optimistic I'll have time to get to that this or next week, although I'm happy for someone to beat me to it.

@oxinabox
Copy link
Collaborator

We should represent unknown sizes with missings, rather than nullables, as they match some of the logic needed. But thatbcan come later

@oxinabox oxinabox self-assigned this Aug 22, 2018
@oxinabox oxinabox removed their assignment Aug 22, 2018
@oxinabox
Copy link
Collaborator

oxinabox commented Aug 22, 2018

I have pushed some commits to this
it is still not working it does take broadcasting seriously. (I think).

So someone else can continue with it from there.

Current error is that it uses JLD.
Cross-ref: #414
Once that is merged, a quick rebase will sort the current error.
Which is all

ERROR: LoadError: LoadError: Failed to precompile JLD 

Now, back to my thesis

@oxinabox
Copy link
Collaborator

can femtocleaner be pointed in this direction?
Since this branch is on @adamryczkowski 's account,
they might have to add Femtoclearner
and configure it to point at this repo.
Following:
https://github.com/JuliaComputing/FemtoCleaner.jl#invoking-femtocleaner

@oxinabox
Copy link
Collaborator

@adamryczkowski can you rebase this? Since #414 is merged.

@adamryczkowski
Copy link
Contributor Author

I see. logistic.jl used to work before your commit, now it throwed the error. Now it is fixed. Thank you.

…tp://www.imagemagick.org

Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules OpenMP
Delegates: bzlib cairo djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png rsvg tiff wmf x xml zlib

Usage: convert [options ...] file [ [options ...] file ...] [options ...] file

Image Settings:
  -adjoin              join images into a single multi-image file
  -affine matrix       affine transform matrix
  -alpha option        activate, deactivate, reset, or set the alpha channel
  -antialias           remove pixel-aliasing
  -authenticate password
                       decipher image with this password
  -attenuate value     lessen (or intensify) when adding noise to an image
  -background color    background color
  -bias value          add bias when convolving an image
  -black-point-compensation
                       use black point compensation
  -blue-primary point  chromaticity blue primary point
  -bordercolor color   border color
  -caption string      assign a caption to an image
  -channel type        apply option to select image channels
  -clip-mask filename  associate a clip mask with the image
  -colors value        preferred number of colors in the image
  -colorspace type     alternate image colorspace
  -comment string      annotate image with comment
  -compose operator    set image composite operator
  -compress type       type of pixel compression when writing the image
  -define format:option
                       define one or more image format options
  -delay value         display the next image after pausing
  -density geometry    horizontal and vertical density of the image
  -depth value         image depth
  -direction type      render text right-to-left or left-to-right
  -display server      get image or font from this X server
  -dispose method      layer disposal method
  -dither method       apply error diffusion to image
  -encoding type       text encoding type
  -endian type         endianness (MSB or LSB) of the image
  -family name         render text with this font family
  -fill color          color to use when filling a graphic primitive
  -filter type         use this filter when resizing an image
  -font name           render text with this font
  -format "string"     output formatted image characteristics
  -fuzz distance       colors within this distance are considered equal
  -gravity type        horizontal and vertical text placement
  -green-primary point chromaticity green primary point
  -intensity method    method to generate intensity value from pixel
  -intent type         type of rendering intent when managing the image color
  -interlace type      type of image interlacing scheme
  -interline-spacing value
                       set the space between two text lines
  -interpolate method  pixel color interpolation method
  -interword-spacing value
                       set the space between two words
  -kerning value       set the space between two letters
  -label string        assign a label to an image
  -limit type value    pixel cache resource limit
  -loop iterations     add Netscape loop extension to your GIF animation
  -mask filename       associate a mask with the image
  -matte               store matte channel if the image has one
  -mattecolor color    frame color
  -moments             report image moments
  -monitor             monitor progress
  -orient type         image orientation
  -page geometry       size and location of an image canvas (setting)
  -ping                efficiently determine image attributes
  -pointsize value     font point size
  -precision value     maximum number of significant digits to print
  -preview type        image preview type
  -quality value       JPEG/MIFF/PNG compression level
  -quiet               suppress all warning messages
  -red-primary point   chromaticity red primary point
  -regard-warnings     pay attention to warning messages
  -remap filename      transform image colors to match this set of colors
  -respect-parentheses settings remain in effect until parenthesis boundary
  -sampling-factor geometry
                       horizontal and vertical sampling factor
  -scene value         image scene number
  -seed value          seed a new sequence of pseudo-random numbers
  -size geometry       width and height of image
  -stretch type        render text with this font stretch
  -stroke color        graphic primitive stroke color
  -strokewidth value   graphic primitive stroke width
  -style type          render text with this font style
  -support factor      resize support: > 1.0 is blurry, < 1.0 is sharp
  -synchronize         synchronize image to storage device
  -taint               declare the image as modified
  -texture filename    name of texture to tile onto the image background
  -tile-offset geometry
                       tile offset
  -treedepth value     color tree depth
  -transparent-color color
                       transparent color
  -undercolor color    annotation bounding box color
  -units type          the units of image resolution
  -verbose             print detailed information about the image
  -view                FlashPix viewing transforms
  -virtual-pixel method
                       virtual pixel access method
  -weight type         render text with this font weight
  -white-point point   chromaticity white point

Image Operators:
  -adaptive-blur geometry
                       adaptively blur pixels; decrease effect near edges
  -adaptive-resize geometry
                       adaptively resize image using 'mesh' interpolation
  -adaptive-sharpen geometry
                       adaptively sharpen pixels; increase effect near edges
  -alpha option        on, activate, off, deactivate, set, opaque, copy
                       transparent, extract, background, or shape
  -annotate geometry text
                       annotate the image with text
  -auto-gamma          automagically adjust gamma level of image
  -auto-level          automagically adjust color levels of image
  -auto-orient         automagically orient (rotate) image
  -bench iterations    measure performance
  -black-threshold value
                       force all pixels below the threshold into black
  -blue-shift factor   simulate a scene at nighttime in the moonlight
  -blur geometry       reduce image noise and reduce detail levels
  -border geometry     surround image with a border of color
  -bordercolor color   border color
  -brightness-contrast geometry
                       improve brightness / contrast of the image
  -canny geometry      detect edges in the image
  -cdl filename        color correct with a color decision list
  -charcoal radius     simulate a charcoal drawing
  -chop geometry       remove pixels from the image interior
  -clamp               keep pixel values in range (0-QuantumRange)
  -clip                clip along the first path from the 8BIM profile
  -clip-path id        clip along a named path from the 8BIM profile
  -colorize value      colorize the image with the fill color
  -color-matrix matrix apply color correction to the image
  -connected-components connectivity
                       connected-components uniquely labeled
  -contrast            enhance or reduce the image contrast
  -contrast-stretch geometry
                       improve contrast by `stretching' the intensity range
  -convolve coefficients
                       apply a convolution kernel to the image
  -cycle amount        cycle the image colormap
  -decipher filename   convert cipher pixels to plain pixels
  -deskew threshold    straighten an image
  -despeckle           reduce the speckles within an image
  -distort method args
                       distort images according to given method ad args
  -draw string         annotate the image with a graphic primitive
  -edge radius         apply a filter to detect edges in the image
  -encipher filename   convert plain pixels to cipher pixels
  -emboss radius       emboss an image
  -enhance             apply a digital filter to enhance a noisy image
  -equalize            perform histogram equalization to an image
  -evaluate operator value
                       evaluate an arithmetic, relational, or logical expression
  -extent geometry     set the image size
  -extract geometry    extract area from image
  -features distance   analyze image features (e.g. contrast, correlation)
  -fft                 implements the discrete Fourier transform (DFT)
  -flip                flip image vertically
  -floodfill geometry color
                       floodfill the image with color
  -flop                flop image horizontally
  -frame geometry      surround image with an ornamental border
  -function name parameters
                       apply function over image values
  -gamma value         level of gamma correction
  -gaussian-blur geometry
                       reduce image noise and reduce detail levels
  -geometry geometry   preferred size or location of the image
  -grayscale method    convert image to grayscale
  -hough-lines geometry
                       identify lines in the image
  -identify            identify the format and characteristics of the image
  -ift                 implements the inverse discrete Fourier transform (DFT)
  -implode amount      implode image pixels about the center
  -interpolative-resize geometry
                       resize image using 'point sampled' interpolation
  -kuwahara geometry   edge preserving noise reduction filter
  -lat geometry        local adaptive thresholding
  -level value         adjust the level of image contrast
  -level-colors color,color
                       level image with the given colors
  -linear-stretch geometry
                       improve contrast by `stretching with saturation'
  -liquid-rescale geometry
                       rescale image with seam-carving
  -magnify             double the size of the image with pixel art scaling
  -mean-shift geometry delineate arbitrarily shaped clusters in the image
  -median geometry     apply a median filter to the image
  -mode geometry       make each pixel the 'predominant color' of the
                       neighborhood
  -modulate value      vary the brightness, saturation, and hue
  -monochrome          transform image to black and white
  -morphology method kernel
                       apply a morphology method to the image
  -motion-blur geometry
                       simulate motion blur
  -negate              replace every pixel with its complementary color
  -noise geometry      add or reduce noise in an image
  -normalize           transform image to span the full range of colors
  -opaque color        change this color to the fill color
  -ordered-dither NxN
                       add a noise pattern to the image with specific
                       amplitudes
  -paint radius        simulate an oil painting
  -perceptible epsilon
                       pixel value less than |epsilon| become epsilon or
                       -epsilon
  -polaroid angle      simulate a Polaroid picture
  -posterize levels    reduce the image to a limited number of color levels
  -profile filename    add, delete, or apply an image profile
  -quantize colorspace reduce colors in this colorspace
  -radial-blur angle   radial blur the image (deprecated use -rotational-blur
  -raise value         lighten/darken image edges to create a 3-D effect
  -random-threshold low,high
                       random threshold the image
  -region geometry     apply options to a portion of the image
  -render              render vector graphics
  -repage geometry     size and location of an image canvas
  -resample geometry   change the resolution of an image
  -resize geometry     resize the image
  -roll geometry       roll an image vertically or horizontally
  -rotate degrees      apply Paeth rotation to the image
  -rotational-blur angle
                       rotational blur the image
  -sample geometry     scale image with pixel sampling
  -scale geometry      scale the image
  -segment values      segment an image
  -selective-blur geometry
                       selectively blur pixels within a contrast threshold
  -sepia-tone threshold
                       simulate a sepia-toned photo
  -set property value  set an image property
  -shade degrees       shade the image using a distant light source
  -shadow geometry     simulate an image shadow
  -sharpen geometry    sharpen the image
  -shave geometry      shave pixels from the image edges
  -shear geometry      slide one edge of the image along the X or Y axis
  -sigmoidal-contrast geometry
                       increase the contrast without saturating highlights or
                       shadows
  -sketch geometry     simulate a pencil sketch
  -solarize threshold  negate all pixels above the threshold level
  -sparse-color method args
                       fill in a image based on a few color points
  -splice geometry     splice the background color into the image
  -spread radius       displace image pixels by a random amount
  -statistic type geometry
                       replace each pixel with corresponding statistic from the
                       neighborhood
  -strip               strip image of all profiles and comments
  -swirl degrees       swirl image pixels about the center
  -threshold value     threshold the image
  -thumbnail geometry  create a thumbnail of the image
  -tile filename       tile image when filling a graphic primitive
  -tint value          tint the image with the fill color
  -transform           affine transform image
  -transparent color   make this color transparent within the image
  -transpose           flip image vertically and rotate 90 degrees
  -transverse          flop image horizontally and rotate 270 degrees
  -trim                trim image edges
  -type type           image type
  -unique-colors       discard all but one of any pixel color
  -unsharp geometry    sharpen the image
  -vignette geometry   soften the edges of the image in vignette style
  -wave geometry       alter an image along a sine wave
  -white-threshold value
                       force all pixels above the threshold into white

Image Sequence Operators:
  -append              append an image sequence
  -clut                apply a color lookup table to the image
  -coalesce            merge a sequence of images
  -combine             combine a sequence of images
  -compare             mathematically and visually annotate the difference between an image and its reconstruction
  -complex operator    perform complex mathematics on an image sequence
  -composite           composite image
  -crop geometry       cut out a rectangular region of the image
  -deconstruct         break down an image sequence into constituent parts
  -evaluate-sequence operator
                       evaluate an arithmetic, relational, or logical expression
  -flatten             flatten a sequence of images
  -fx expression       apply mathematical expression to an image channel(s)
  -hald-clut           apply a Hald color lookup table to the image
  -layers method       optimize, merge, or compare image layers
  -morph value         morph an image sequence
  -mosaic              create a mosaic from an image sequence
  -poly terms          build a polynomial from the image sequence and the corresponding
                       terms (coefficients and degree pairs).
  -print string        interpret string and print to console
  -process arguments   process the image with a custom image filter
  -separate            separate an image channel into a grayscale image
  -smush geometry      smush an image sequence together
  -write filename      write images to this file

Image Stack Operators:
  -clone indexes       clone an image
  -delete indexes      delete the image from the image sequence
  -duplicate count,indexes
                       duplicate an image one or more times
  -insert index        insert last image into the image sequence
  -reverse             reverse image sequence
  -swap indexes        swap two images in the image sequence

Miscellaneous Options:
  -debug events        display copious debugging information
  -distribute-cache port
                       distributed pixel cache spanning one or more servers
  -help                print program options
  -list type           print a list of supported option arguments
  -log format          format of debugging information
  -version             print version information

By default, the image format of `file' is determined by its magic
number.  To specify a particular image format, precede the filename
with an image format name and a colon (i.e. ps:image) or specify the
image type as the filename suffix (i.e. image.ps).  Specify 'file' as
'-' for standard input or output. an object of type String to an object of type Array{UInt8,1}
@adamryczkowski
Copy link
Contributor Author

I think I stumped upon another non-trivial issue. To reproduce:

julia> import Pkg
julia> Pkg.add(Pkg.PackageSpec(url="https://github.com/adamryczkowski/TensorFlow.jl"))
julia> import TensorFlow
julia> filename = joinpath(mktempdir(), "tfrecords")

julia> TensorFlow.io.RecordWriter(filename) do writer
    write(writer, UInt8[1,2])
    write(writer, "hi")
end

julia> reader = TensorFlow.io.RecordIterator(filename)
julia> records = collect(reader)
ERROR: MethodError: Cannot `convert` an object of type String to an object of type Array{UInt8,1}
Closest candidates are:
  convert(::Type{Array{UInt8,1}}, ::PyCall.PyObject) at /home/adam/.julia/packages/PyCall/akNFy/src/conversions.jl:118
  convert(::Type{T<:Array}, ::AbstractArray) where T<:Array at array.jl:489
  convert(::Type{T<:AbstractArray}, ::T<:AbstractArray) where T<:AbstractArray at abstractarray.jl:14
  ...
Stacktrace:
 [1] convert(::Type{Nullables.Nullable{Array{UInt8,1}}}, ::Nullables.Nullable{String}) at /home/adam/.julia/packages/Nullables/3JHB8/src/nullable.jl:57
 [2] TensorFlow.io.TFRecord.RecordIteratorState(::Nullables.Nullable{String}) at /home/adam/tmp/TensorFlow.jl/src/io/tfrecord.jl:57
 [3] macro expansion at /home/adam/tmp/TensorFlow.jl/src/TensorFlow.jl:179 [inlined]
 [4] _next(::TensorFlow.io.TFRecord.RecordIterator) at /home/adam/tmp/TensorFlow.jl/src/io/tfrecord.jl:76
 [5] iterate at /home/adam/tmp/TensorFlow.jl/src/io/tfrecord.jl:94 [inlined]
 [6] _collect(::UnitRange{Int64}, ::TensorFlow.io.TFRecord.RecordIterator, ::Base.HasEltype, ::Base.SizeUnknown) at ./array.jl:568
 [7] collect(::TensorFlow.io.TFRecord.RecordIterator) at ./array.jl:557
 [8] top-level scope at none:0

One obvious solution is to overload convert so the following code succeeds:

convert(Vector{UInt8}, "Hello world")

But obviously, this change will interfere with the language, and that is why this conversion is not built-in. I have read the discussion JuliaLang/julia#24388 but still I am not sure, how to solve this problem cleanly.

@oxinabox
Copy link
Collaborator

In this case, how to fix that doesn't matter.
Because now julia's iteration protocol and python's are in sync -- no need to buffer on item ahead anymore
Which means we can just delete the RecordIteratorState type.

Which looks something like:

struct RecordIterator
    pyo::Distributed.Future
end


"""
    `RecordIterator(path)`
Returns a Julia iterator that returns the records in the TF Record file
at `path` as `Vector{UInt8}` objects.
"""
function RecordIterator(path::AbstractString)
    pyo = @tf.py_proc py_tf[][:python_io][:tf_record_iterator]($path)
    RecordIterator(pyo)
end

function _next(iter::RecordIterator)
    try
        @static if PyCall.pyversion >= v"3.0.0"
            fetch(@tf.py_proc $(iter.pyo)[:__next__]())
        else
            #Python 2
            fetch(@tf.py_proc $(iter.pyo)[:next]())
        end
        
    catch err
        if isa(err, Distributed.RemoteException) && isa(err.captured.ex, PyCall.PyError)
            # Only catch it, if it could be an  StopIteration exception thrown in python
            # which signifies the end of iteration being reached normally
            nothing # signal to stop julia iteration
        else
            rethrow(err)
        end
    end
end

function Base.iterate(iter::RecordIterator, state=iter)
	record = _next(iter)
	if record isa nothing
		nothing # end iteration
	else
		(record, iter)
	end
end

I don't have a working copy of libtensorflow.so right now so can't test.
But that is the gist of it.

May also need to change the eltype of the iterator itself, or remove the declaration of the eltype.

@adamryczkowski
Copy link
Contributor Author

Many thanks!

@adamryczkowski
Copy link
Contributor Author

I do not understand, how your code would fix the need to have an implicit conversion from String to Array{UInt8,1}.

The error remains exactly the same. But I found a very simple fix. Just convert whatever Python returned into a byte array.

Now I found an interesting problem (and already fixed it) - I do not have Julia 0.6.x to test it, but I have a reason to believe, that polygamma and zeta could not possibly have worked due to a bug. These functions were not exported (i.e. defined) by the TensorFlow, so either the math.jl test failed (as it did in Julia 1.0.0) or Julia cheated and silently used the native version of those functions.

I have inserted

for jl_func_name in [
    :polygamma,
    :zeta]
    @eval @define_binary SpecialFunctions.$jl_func_name Ops.$jl_func_name
end

to ops/math.jl.

I still have 1-2 hours of time to work on the porting. I will tell you, @oxinabox when I am done for today.

@@ -2,6 +2,7 @@

module Ops
import TensorFlow
import SpecialFunctions
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't directly edit this file BTW.
It is automatically generated,

But this change does nothing anyway, so it can happily be removed.

@oxinabox
Copy link
Collaborator

oxinabox commented Aug 28, 2018

Now I found an interesting problem (and already fixed it) - I do not have Julia 0.6.x to test it, but I have a reason to believe, that polygamma and zeta could not possibly have worked due to a bug. These functions were not exported (i.e. defined) by the TensorFlow, so either the math.jl test failed (as it did in Julia 1.0.0) or Julia cheated and silently used the native version of those functions.

Yeah, I see what you mean.
I don't know why it works in the tests in julia 0.6.
But it does (I just tested), and it even gives the right answer. (I added a println).
But the code to calculate that doesn't even exist in the julia 0.6 source.
So something funky is going on.

Very mysterious but ultimately not a concern

I do not understand, how your code would fix the need to have an implicit conversion from String to Array{UInt8,1}.
The error remains exactly the same. But I found a very simple fix. Just convert whatever Python returned into a byte array.

huh, I must have misread which line the error was on, I thought it was in the constructor for the RecordIteratorState
But yes, in general one does need to convert what python returns.

@adamryczkowski
Copy link
Contributor Author

@malmaud I found a bug in the existing TensorFlow code which apparently slipped the unit tests. You might want to take a look, because there may be more potential bugs that were not the detected by the unit tests as well. The bug is described in my previous comment.

@adamryczkowski
Copy link
Contributor Author

Suppose I have

a = TensorFlow.constant(10)

How can I get the number 10 out of a without calling TensorFlow (i.e. without calling run(TensorFlow.Session(), a)?

Is it possible?

I believe I need it to solve a following bug

julia> using TensorFlow
julia> import LinearAlgebra

julia> a = TensorFlow.constant(LinearAlgebra.Diagonal(ones(5,5)))
julia> idxs = TensorFlow.constant(map(Int64, [1,1,2,3,1]))
julia> n = TensorFlow.constant(Int32(3))
julia> d = unsorted_segment_sum(a, idxs, n);
julia> TensorFlow.get_shape(d)
ERROR: MethodError: no method matching Array(::TensorFlow.RawTensor)
Closest candidates are:
  Array(::LinearAlgebra.SymTridiagonal) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/LinearAlgebra/src/tridiag.jl:111
  Array(::LinearAlgebra.Tridiagonal) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/LinearAlgebra/src/tridiag.jl:518
  Array(::LinearAlgebra.AbstractTriangular) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/LinearAlgebra/src/triangular.jl:106
  ...
Stacktrace:
 [1] load_const(::Operation) at /home/adam/tmp/TensorFlow.jl/src/shape_inference.jl:260
 [2] load_const at /home/adam/tmp/TensorFlow.jl/src/shape_inference.jl:331 [inlined]
 [3] (::getfield(TensorFlow.ShapeInference, Symbol("##42#43")))(::Operation) at /home/adam/tmp/TensorFlow.jl/src/shape_inference.jl:425
 [4] _get_shape(::Tensor{Float64}) at /home/adam/tmp/TensorFlow.jl/src/shape_inference.jl:107
 [5] get_shape(::Tensor{Float64}) at /home/adam/tmp/TensorFlow.jl/src/shape_inference.jl:78
 [6] top-level scope at none:0

Or maybe the solution is different. That is all I can do for today.

@oxinabox
Copy link
Collaborator

The normal way to do that is to use load_const,
which is the function where your error occurs.
There are lower level ways to do it like

julia> x = constant(10)
<Tensor Const_2:1 shape=() dtype=Int64>

julia> x.op["value"].tensor.int64_val
1-element Array{Int64,1}:
 10

In anycase
That error to me looks like the use of a conversion-constructor that has plagued use constantly in this PR.

In particular,
I expect that

value = Array(tf.load_proto(get_def(op).attr["value"]))

Should be changed to

raw_value = tf.load_proto(get_def(op).attr["value"])
value  = convert(Array, raw_value)

Not tested, but that is my guess..

@malmaud
Copy link
Owner

malmaud commented Aug 28, 2018

wrt how we'll merge this, my opinion is just a merge commit. Then everyone's contributions are clearly recorded but the main branch of the history still doesn't have a plethora of intermediate commits.

@malmaud
Copy link
Owner

malmaud commented Aug 29, 2018

Tests pass, so I'm going to merge this now. Thanks everyone for putting this together!

@malmaud malmaud merged commit ce253c7 into malmaud:master Aug 29, 2018
@adamryczkowski
Copy link
Contributor Author

adamryczkowski commented Aug 29, 2018

I can confirm, that all tests pass in all my systems as well. Just make sure, you have proper Python installed with the TensorFlow library accessible from it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants