Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

RFC: add multimedia I/O mechanism (display, mm_write, and friends); fixes #3817 #3932

Merged
merged 2 commits into from

9 participants

@stevengj
Collaborator

This patch addresses #3817 to provide a general mechanism in Julia to display objects via rich multimedia representations, via three components:

  • A function display(x) to request the richest available multimedia display of a Julia object x (with a text/plain fallback).
  • Overloading writemime allows one to indicate arbitrary multimedia representations (keyed by standard MIME types) of user-defined types.
  • Multimedia-capable display backends may be registered by subclassing a generic Display type and pushing them onto a stack of display backends via pushdisplay.

This design has gone through several iterations since the inception of #3817, and is in part inspired by the design of IPython's custom display logic. It is especially critical for the development of IJulia, where I have tested this technique and found that it works very well.

The main difference from my most recent proposal in #3817 is that I generalized the mechanism to be extensible to arbitrary MIME types without modifying Base, using a parametric singleton-type technique similar to @StefanKarpinski's beautiful MathConst trick in order to allow Julia to dispatch on the MIME type. For example, if I have a type MyImage that I know how to write as PNG, I can simply define a new writemime method:

import Base.writemime
writemime(stream, ::@MIME("image/png"), x::MyImage) = ...write x as PNG to stream...

and any MyImage object will automatically display as a PNG image in IJulia or in any other display device supporting image/png.

See the changes to doc/stdlib/base.rst for the complete documentation.

I also added and documented a base64(x) function to base64-encode binary data, as this is generically useful for sending MIME data to backends over string-based transport protocols (and is essential for IJulia), based on some code by @StefanKarpinski.

cc: @JeffBezanson, @ViralShah, @timholy, @fperez, @loladiro

@stevengj
Collaborator

There is also a redisplay(x) function that defaults to simply calling display, but which a backend may optionally override in order to modify an existing display for x (as opposed to, for example, opening a new image window). IJulia uses this to defer display until an entire input cell is executed.

This is useful for Pylab/Matlab-like stateful plotting modules in which the a plot is created (e.g. by plot) and then modified many times (e.g. by xlabel, title, etcetera); each of these functions would simply call redisplay.

@stevengj stevengj referenced this pull request from a commit in JuliaLang/IJulia.jl
@stevengj stevengj update to latest Multimedia API from JuliaLang/julia#3932 114d47d
@ViralBShah
Owner

I like the simplicity of this approach. @dcjones - IIRC, you have a base64 function in Codecs.jl. Could you take a look?

@timholy
Collaborator

Let me test my understanding on a concrete example. Now we have two ways of showing profiling data: as text (using Profile.print()) and as an image (defined in ProfileView.jl). Let's say I've gotten my profiling data all wrapped up in a ProfileData type (which doesn't currently exist, but it easily could), with that variable called p.

Now IIUC I would register my two methods something like this:

mm_write(io, ::@MIME("text/plain"), p::ProfileData) = Profile.print(io, p)
mm_write(io, ::@MIME("image/png"), p::ProfileData) = ProfileView.view(io, p)

Really the last one pops up a GUI, so "image/png" doesn't seem quite correct. How does one distinguish between popping up a Tk GUI vs. base64-encoding at a bytestream to send to IPython? That's the job of io? What if I don't supply io, and I haven't yet popped up any Tk windows?

@stevengj
Collaborator

@timholy, mm_write does not "pop up a GUI". It simply writes p in image/png format to io, which is just an ordinary binary stream (io::IO). So, you'd probably want it to call some kind of ProfileView.export_png(io, p) function.

The GUI is "popped up" (if at all) during display(p) by an instance of a Display subtype which has been pushed onto the display stack. For instance, IJulia pushes its own InlineDisplay instance onto the display stack. When display(p) is called, it invokes display(d, p) in turn for each display d in the stack until one succeeds. display(d, p), in turn, calls mm_writable(mime, typeof(p)) (true if the corresponding mm_write method exists) for each MIME type mime that d knows how to display. So, for example, the IJulia InlineDisplay knows how to display any object that is writable as image/png, image/jpeg, image/svg+xml, text/html, application/x-latex, or text/plain, and will succeed for any p such that mm_write is defined for one of these types.

For a supported MIME type, display(d, p) will normally call mm_repr(mime, p) or mm_string_repr(mime, p), which allocates a memory buffer (IOBuffer) and calls mm_write(io, mime, p) to write p to that buffer in the requested format. Given that data, the display is then responsible for displaying it (by popping up a GUI, writing it to disk, tossing it in the trash, or whatever it wants to do ... in IJulia's case, we send it to the IPython front-end, which sends it to the browser).

The key is that we separate multimedia export of Julia types from multimedia display of the exported data.

PS. Also, you don't have to define a text/plain mm_write, since a fallback text/plain mm_write is already defined for all types, which simply calls repl_show.

@timholy
Collaborator

OK, I think I see now. Let's say a user loads the Images package (which does not handle any kind of graphical output, it's simply image types, I/O, and algorithms). The Images package should presumably define mm_write for image/png and image/jpg (currently we use ImageMagick to handle that). Then, if the user is working from IJulia, display(img) will route it to IJulia, because IJulia has registered a Display.

But let's say instead I'm working from IJulia and then load ImageView. (Let's say I love IJulia as a "better REPL" but need the features and/or performance of ImageView for my specific task.) Are you envisioning that I'll short-circuit all this with my specific display(img::Image) function that takes precedence over yours? Anyone who wants to send an image back to IJulia while also using ImageView can do so by saying display(disp_ijulia, img). Honestly, I'm fine with this as a solution, at least until we discover that it causes problems.

Alternatively, are you hoping ImageView will integrate more closely with this mechanism? If so, presumably it should push a Display instance onto the stack when it loads, even though it hasn't yet opened a window, so that when a user says display(img) it will default to the ImageView one. So in some senses, the ImageView Display instance has to be a mini window manager? There may be a few other interesting details related to display state, which are briefly described here and which may be clearer if you refer to the definition of an ImageCanvas

@stevengj
Collaborator

@timholy, what I'm hoping is that ImageView will integrate with this mechanism, by defining a Display subtype that knows how to display image/png etcetera. You may or may not want to push this onto the display stack when ImageView is loaded, but if you do then it will be used to display those image types instead of IJulia (if IJulia is being used). How you manage your window state is up to you — whether you put all of the state into a Display subtype, or you continue to maintain global state as now and your Display subtype is just a dispatch hook that turns around and calls your existing display code.

You are free, of course, to define additional display-like functions that provide more control over the display; display is just the baseline.

Furthermore, the return value of display is implementation-defined (you throw a MethodError exception if you are handed a type you can't display), so if you want to return some kind of display "handle" you are free to do so, and the user can take advantage of this if they know they are using your package.

Finally, if you want provide even richer display of a certain Julia type (say you provide interactive editing, not just display, of a MyImage type), you can do so: just provide a display(d::MyDisplay, x::MyImage) method to do whatever you want for that specific type, and the Julia's method-dispatch will call it automatically whenever display(x) is invoked for x::MyImage.

The whole point of putting this into Julia Base is that both multimedia export and multimedia backends are basic functionalities that can be provided in lots of ways, by lots of packages — this is not something that can or should be limited to IJulia!

@timholy
Collaborator

OK, I think I'm slowly catching on. My current thought is that perhaps Tk and Gtk are perhaps the right places to put the parts dealing with low-level window state management.

Anyway, I'm good to merge on this. We'll figure any issues out as they come up.

@ivarne
Collaborator

Should the mm_write function get a hint about the display size? For non vector images it makes a huge difference, and I think that the display should be able to ask for a specific hight and/or width, even if the mm_write implementation might ignore the hint.

Does this interface support display devices might start to show the first frames of an animation before the last framme is generated and mm_write returns?

@stevengj
Collaborator

@ivarne, with regards to the first question, my inclination is to think of that as a more application-specific thing; e.g. the default height and width you might want for plotting output would be very different from the default height and width for emoticon output, and since the display knows nothing about the source of the data it doesn't seem like the best place to put that information.

With regards to the second question, it should be perfectly possible in Julia for the display to read and display the output of mm_write asynchronously in a coroutine (thanks to Julia's asynchronous I/O support) in order to display the data in a streaming fashion. It could even discard the output of mm_write as it displays it, so that the whole animation need not be stored in memory at once. @loladiro can probably comment more intelligently on how asynchronous I/O would fit in here.

@Keno
Owner

For asynchronous I/O all that's needed is some kind of IO object that can handle the data and block the writing task when appropriate.

@stevengj
Collaborator

Note that the IPython protocol does incorporate a metadata field that can be used to pass arbitrary data along with the MIME data, and their metadata field is used to pass image-size hints (see ipython/ipython#3190), but it is used in the opposite direction, as I understand it (the data source sends the full image but sometimes hints that it should be shown as a different size, e.g. a thumbnail). However, even though they support arbitrary metadata, these thumbnailing hints are still essentially the only thing their metadata is used for. It seemed to me that this was a problematic design, and in any case there should probably be a more global way to indicate thumbnailing preferences for large images.

@JeffBezanson
Owner

Maybe it should be called writemm or writemime, after writecsv?

@ViralBShah
Owner

+1 for writemime.

@stevengj
Collaborator

Okay, so mm_write, mm_repr, mm_string_repr, and mm_writable become writemime, reprmime, stringmime, and mimewritable, respectively?

(I agree that "mime" is more readable, although it may be technically incorrect. We are using MIME types (or rather, "Internet media types"), but we are not writing MIME email attachments. I don't really care if you don't, though.)

@stevengj
Collaborator

Okay, renamed, and also renamed {push/pop}_display to {push/pop}display

@stevengj
Collaborator

Okay to merge?

@Keno
Owner

Fine with me

@StefanKarpinski

@JeffBezanson and I have both been away over the weekend and are still catching up, could you wait a bit so we can review and comment. Sorry for the delay!

@stevengj
Collaborator

@StefanKarpinski, no problem.

@Carreau

Hi,

Sorry I'm not a julia expert and I have a few questions.

I didn't totally got how is the richest representation decided in display.

Back to image/png, image/jpeg, image/svg+xml, text/html, application/x-latex.
If my object know how to write itself as image/png, image/svg+xml, application/x-latex.
Would this mean that only the first match will be sent to the fontend ? or the three ?

The for each display d in the stack **until** one succeeds make me think that only the first will.

In this case (only the first) how will this apply when multiple frontend with different display capability are hooked to the same kernel ?

Note that the IPython protocol does incorporate a metadata field that can be used to pass arbitrary data along with the MIME data, and their metadata field is used to pass image-size hints (see ipython/ipython#3190), but it is used in the opposite direction, as I understand it (the data source sends the full image but sometimes hints that it should be shown as a different size, e.g. a thumbnail). However, even though they support arbitrary metadata, these thumbnailing hints are still essentially the only thing their metadata is used for. It seemed to me that this was a problematic design, and in any case there should probably be a more global way to indicate thumbnailing preferences for large images.

The metadata will mainly be use for javascript plugin to receive extra information about the mimetype they are supposed to handle. Right now the only javascript widget we have is the one that deal with image and resize, and so get only width and height metadatafield, but if the image was read from disk you could send along permission, last modified, size on disk... on my case I have some biological images I can send a scale with it.

Brian did some D3js network graph animation using json representation and some 3d vtk in browser, obviously you want to send the pure data representing to the javascript, but all the unrelevant-meta-information that might be plugin specific like the speed of animation or the original orientation of the 3D model have their place in metadata.

The point is that as data are stored in notebook format, they should be handler independent, hence the optional metadata field.

Hope this clarify what I understood of how metadata will be used, and not only for thumbnailing.

@stevengj
Collaborator

@Carreau, whichever frontend that is called has access to all of the available representations; it can call as many mimewrite functions as it wants. For example, the IJulia InlineDisplay sends multiple representations to IPython in a JSON message. The loop you were referring to means that the a given object is sent to only one display. For example, if someone pushes a specialized display for images, then images will be shown in that viewer instead of in the IJulia notebook.

I don't understand why a separate metadata field is needed for the uses you mentioned, as opposed to embedding the metadata in the Javascript or elsewhere in the HTML DOM (with Javascript embedded via <script>).

@Carreau

@Carreau, whichever frontend that is called has access to all of the available representations; it can call as many mimewrite functions as it wants. For example, the IJulia InlineDisplay sends multiple representations to IPython in a JSON message.

Ok, I'm starting to get it. I still have difficulties to separate the julia from the IJulia layer, but it makes sens.

I don't understand why a separate metadata field is needed for the uses you mentioned, as opposed to embedding the metadata in the Javascript or elsewhere in the HTML DOM (with Javascript embedded via <script>).

This is because we try as much as possible to avoid making assumption that the frontend will be javascript/html.
.ipynb format does not assume it will be attached to the DOM, nothing from the DOM is actually stored when saving. the Javascript "Cell" object receive exactly the same data at load time that when you executed the notebook. right now the .ipynb json structure is on JS side, but we plan on moving it to the python (server not kernel) side, so that you could actually launch some computation, close your browser, come back

For example Emacs-IPython-Notebook speek with the server through websocket, also have a dispatch on mime-type it might interpreat the size attribute of the metadata to scale image (maybe it does, or not, I have no clues), but still it needs to be able to store it as JSON, as the .ipynb file it saves might be loaded in the html notebook and vice-versa.

Another type of "frontend" is be nbconvert. It actually read the json file, which include mimetype/data/metadata and use it to build a converted document like PDF, where there is no DOM. Metadata can be use to scale image, or assign a \ref{<somename>} to this figure. In the same way, nbconvert could be use to generate ipynb file in a headless manner, and metadata need to be stored in the fileformat for other frontend to be read.

We could have send everything in the following form

"mimetype-key" : {
  metadata :<whatever>, 
  content: <actual content> 
}

But this would have involve significant refactor on one side, also, json does not support binary data which should allow to keep the content binary as long as possible.

You could, in some ways, see metadata of as exif, but for all mime-types, even thoses that don't support it.

@stevengj
Collaborator

I'm still confused about the justification for your metadata:

  • Having a separate metadata makes it much harder to send the data to anything that is not IPython (or IPython-aware) without discarding the metadata, since the metadata is not part of the respective MIME format.
  • Any program that would want to make use of the metadata would have to know about the specfic program that created the data (both because the metadata is not part of the MIME data and because its contents are not defined by IPython either). But in that case, you could use almost any parseable mechanism you want to embed the metadata directly in the data. For example, specially formatted comments in HTML/Javascript or EXIF-like tags in an image would work just as well—better, in fact, because of the first point above.
@Carreau

Ok, let's step back I think we probably miss understand each other.

right now we have (form [here]):

content = {
     ...
    'data' : dict, # key are mimetype
    'metadata' : dict #key are mimetype
}

You would like something more like

content = {
     ...
    'dataAndMetaData' : {
         'mimetype' : value
     }
}

Where value is some way of carrying data and metadata. Right ? One mimetype -> one object ?
I don't disagree with you, I think we went the other route for compatibility reason, as it is much more easy to add a field than to modify completely message spec.

Any program that would want to make use of the metadata would have to know about the specfic program that created the data (both because the metadata is not part of the MIME data and because its contents are not defined by IPython either)

Yes, in this should mostly be true in any metada. Nobody should expect metadata to be there, they might not be.
We will though document best practice of what people might expect in metadata. For images those might be size, cell metadata will have an proposed name and tags list which would respectively contain a string and a list of string.

But in that case, you could use almost any parseable mechanism you want to embed the metadata directly in the data. For example, specially formatted comments in HTML/Javascript or EXIF-like tags in an image would work just as well—better, in fact, because of the first point above.

That would be creating yet another mimetype for each existing mimetype. For me, the data of the mimetype should be untouched as many library already now how to process those file. Moreover, embedding the metadata into the data themselve would require to write a extractor for those metadata for each mimetype. Consider also the fact that each embeded data could be extracted from the ipynb file. What assure you that the clever embeding you did in some mimetype will not choke another software ? Sure it might work in html or javascript as you know the comments, but what about application/octet-stream ?

@stevengj
Collaborator

No. I just want you to deprecate the metadata field, and leave data as-is (a dictionary from MIME type to corresponding data). (Or actually, I don't care if you keep the metadata field or not; we can always just pass empty metadata.) In any case, the wire format of IPython is not the topic here.

If a particular application needs to embed some additional meta-information in an application-specific way, that will be read and used by an application-specific front-end, it can do that by embedding it within the existing data as I suggested.

That would be creating yet another mimetype for each existing mimetype.

No it wouldn't, because my suggestions (comments in Javascript source or EXIF-like tags in images) utilize existing metadata mechanisms in those formats that would simply be ignored by front-ends that don't know to look for the application-specific metadata. (Alternatively, in a binary format with predictable/detectable length but no designated metadata fields, you could just append application-specific binary metadata beyond the end of the MIME binary data, which would be ignored by readers that didn't know to look for it. But most formats these days already have ways to embed arbitrary metadata within them.)

Yes, it would be hard to embed metadata in application/octet-stream. But will you really lose any sleep over this? Give me a concrete IPython situation that would need application/octet-stream + metadata, and wouldn't simply use application/x-mycustomtype.

@stevengj stevengj referenced this pull request in stevengj/PyPlot.jl
Closed

Replace with PyCall-based version? #8

@JeffBezanson
Owner

I like it.
Looks like the functions were not renamed in the exports list?

@stevengj
Collaborator

@JeffBezanson, fixed the exports & flushing. I'm actually not sure in retrospect why I bothered flushing, or why I ignored errors from the flush.. Maybe I should just omit the flush?

@stevengj
Collaborator

Not sure I understand the Travis failure; at first glance, it has nothing to do with this patch...

@JeffBezanson
Owner

I'm ready to merge this. @StefanKarpinski ?

@staticfloat
Owner

@stevengj We've got a couple bugs that are floating around, some of which seem to only manifest on Travis builds, others which intermittently show up on our machines as well. In general, if only one of the builds fail (e.g. gcc fails while the clang build passes), it's one of those problems. If both fail, then you should start to suspect your changes.

I've restarted the Travis build, it should show up green if we're lucky. (How I hate intermittent failures)

@JeffBezanson JeffBezanson merged commit ad1b0de into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
137 base/base64.jl
@@ -0,0 +1,137 @@
+module Base64
+import Base: read, write, close
+export Base64Pipe, base64
+
+# Base64Pipe is a pipe-like IO object, which converts writes (and
+# someday reads?) into base64 encoded (decoded) data send to a stream.
+# (You must close the pipe to complete the encode, separate from
+# closing the target stream). We also have a function base64(f,
+# args...) which works like sprint except that it produces
+# base64-encoded data, along with base64(args...) which is equivalent
+# to base64(write, args...), to return base64 strings.
+
+#############################################################################
+
+type Base64Pipe <: IO
+ io::IO
+ # writing works in groups of 3, so we need to cache last two bytes written
+ b0::Uint8
+ b1::Uint8
+ nb::Uint8 # number of bytes in cache: 0, 1, or 2
+
+ function Base64Pipe(io::IO)
+ b = new(io,0,0,0)
+ finalizer(b, close)
+ return b
+ end
+end
+
+#############################################################################
+
+# Based on code by Stefan Karpinski from https://github.com/hackerschool/WebSockets.jl (distributed under the same MIT license as Julia)
+
+const b64chars = ['A':'Z','a':'z','0':'9','+','/']
+
+function b64(x::Uint8, y::Uint8, z::Uint8)
+ n = int(x)<<16 | int(y)<<8 | int(z)
+ b64chars[(n >> 18) + 1],
+ b64chars[(n >> 12) & 0b111111 + 1],
+ b64chars[(n >> 6) & 0b111111 + 1],
+ b64chars[(n ) & 0b111111 + 1]
+end
+
+function b64(x::Uint8, y::Uint8)
+ a, b, c = b64(x, y, 0x0)
+ a, b, c, '='
+end
+
+function b64(x::Uint8)
+ a, b = b64(x, 0x0, 0x0)
+ a, b, '=', '='
+end
+
+#############################################################################
+
+function write(b::Base64Pipe, x::AbstractVector{Uint8})
+ n = length(x)
+ s = 1 # starting index
+ # finish any cached data to write:
+ if b.nb == 1
+ if n >= 2
+ write(b.io, b64(b.b0, x[1], x[2])...)
+ s = 3
+ elseif n == 1
+ b.b1 = x[1]
+ b.nb = 2
+ return
+ else
+ return
+ end
+ elseif b.nb == 2
+ if n >= 1
+ write(b.io, b64(b.b0, b.b1, x[1])...)
+ s = 2
+ else
+ return
+ end
+ end
+ # write all groups of three bytes:
+ while s + 2 <= n
+ write(b.io, b64(x[s], x[s+1], x[s+2])...)
+ s += 3
+ end
+ # cache any leftover bytes:
+ if s + 1 == n
+ b.b0 = x[s]
+ b.b1 = x[s+1]
+ b.nb = 2
+ elseif s == n
+ b.b0 = x[s]
+ b.nb = 1
+ else
+ b.nb = 0
+ end
+end
+
+function write(b::Base64Pipe, x::Uint8)
+ if b.nb == 0
+ b.b0 = x
+ b.nb = 1
+ elseif b.nb == 1
+ b.b1 = x
+ b.nb = 2
+ else
+ write(b.io, b64(b.b0,b.b1,x)...)
+ b.nb = 0
+ end
+end
+
+function close(b::Base64Pipe)
+ if b.nb > 0
+ # write leftover bytes + padding
+ if b.nb == 1
+ write(b.io, b64(b.b0)...)
+ else # b.nb == 2
+ write(b.io, b64(b.b0, b.b1)...)
+ end
+ b.nb = 0
+ end
+end
+
+# like sprint, but returns base64 string
+function base64(f::Function, args...)
+ s = IOBuffer()
+ b = Base64Pipe(s)
+ f(b, args...)
+ close(b)
+ takebuf_string(s)
+end
+base64(x...) = base64(write, x...)
+
+#############################################################################
+
+# read(b::Base64Pipe, ::Type{Uint8}) = # TODO: decode base64
+
+#############################################################################
+
+end # module
View
18 base/exports.jl
@@ -726,6 +726,8 @@ export
# strings and text output
ascii,
base,
+ base64,
+ Base64Pipe,
beginswith,
bin,
bits,
@@ -1119,6 +1121,22 @@ export
wait,
workers,
+# multimedia I/O
+ Display,
+ display,
+ displayable,
+ TextDisplay,
+ istext,
+ MIME,
+ @MIME,
+ reprmime,
+ stringmime,
+ writemime,
+ mimewritable,
+ popdisplay,
+ pushdisplay,
+ redisplay,
+
# distributed arrays
dfill,
distribute,
View
214 base/multimedia.jl
@@ -0,0 +1,214 @@
+module Multimedia
+
+export Display, display, pushdisplay, popdisplay, displayable, redisplay,
+ MIME, @MIME, writemime, reprmime, stringmime, istext,
+ mimewritable, TextDisplay, reinit_displays
+
+###########################################################################
+# We define a singleton type MIME{mime symbol} for each MIME type, so
+# that Julia's dispatch and overloading mechanisms can be used to
+# dispatch writemime and to add conversions for new types.
+
+immutable MIME{mime} end
+
+import Base: show, string, convert
+MIME(s) = MIME{symbol(s)}()
+show{mime}(io::IO, ::MIME{mime}) = print(io, "MIME type ", string(mime))
+string{mime}(::MIME{mime}) = string(mime)
+
+# needs to be a macro so that we can use ::@mime(s) in type declarations
+macro MIME(s)
+ quote
+ MIME{symbol($s)}
+ end
+end
+
+###########################################################################
+# For any type T one can define writemime(io, ::@MIME(mime), x::T) = ...
+# in order to provide a way to export T as a given mime type.
+
+# We provide a fallback text/plain representation of any type:
+writemime(io, ::@MIME("text/plain"), x) = repl_show(io, x)
+
+mimewritable{mime}(::MIME{mime}, T::Type) =
+ method_exists(writemime, (IO, MIME{mime}, T))
+
+# it is convenient to accept strings instead of ::MIME
+writemime(io, m::String, x) = writemime(io, MIME(m), x)
+mimewritable(m::String, T::Type) = mimewritable(MIME(m), T)
+
+###########################################################################
+# MIME types are assumed to be binary data except for a set of types known
+# to be text data (possibly Unicode). istext(m) returns whether
+# m::MIME is text data, and reprmime(m, x) returns x written to either
+# a string (for text m::MIME) or a Vector{Uint8} (for binary m::MIME),
+# assuming the corresponding write_mime method exists. stringmime
+# is like reprmime except that it always returns a string, which in the
+# case of binary data is Base64-encoded.
+#
+# Also, if reprmime is passed a String for a text type or Vector{Uint8} for
+# a binary type, the argument is assumed to already be in the corresponding
+# format and is returned unmodified. This is useful so that raw data can be
+# passed to display(m::MIME, x).
+
+for mime in ["text/cmd", "text/css", "text/csv", "text/html", "text/javascript", "text/plain", "text/vcard", "text/xml", "application/atom+xml", "application/ecmascript", "application/json", "application/rdf+xml", "application/rss+xml", "application/xml-dtd", "application/postscript", "image/svg+xml", "application/x-latex", "application/xhtml+xml", "application/javascript", "application/xml", "model/x3d+xml", "model/x3d+vrml", "model/vrml"]
+ @eval begin
+ istext(::@MIME($mime)) = true
+ reprmime(m::@MIME($mime), x::String) = x
+ reprmime(m::@MIME($mime), x) = sprint(writemime, m, x)
+ stringmime(m::@MIME($mime), x) = reprmime(m, x)
+ # avoid method ambiguities with definitions below:
+ # (Q: should we treat Vector{Uint8} as a bytestring?)
+ reprmime(m::@MIME($mime), x::Vector{Uint8}) = sprint(writemime, m, x)
+ stringmime(m::@MIME($mime), x::Vector{Uint8}) = reprmime(m, x)
+ end
+end
+
+istext(::MIME) = false
+function reprmime(m::MIME, x)
+ s = IOBuffer()
+ writemime(s, m, x)
+ takebuf_array(s)
+end
+reprmime(m::MIME, x::Vector{Uint8}) = x
+stringmime(m::MIME, x) = base64(writemime, m, x)
+stringmime(m::MIME, x::Vector{Uint8}) = base64(write, x)
+
+# it is convenient to accept strings instead of ::MIME
+istext(m::String) = istext(MIME(m))
+reprmime(m::String, x) = reprmime(MIME(m), x)
+stringmime(m::String, x) = stringmime(MIME(m), x)
+
+###########################################################################
+# We have an abstract Display class that can be subclassed in order to
+# define new rich-display output devices. A typical subclass should
+# overload display(d::Display, m::MIME, x) for supported MIME types m,
+# (typically using reprmime or stringmime to get the MIME
+# representation of x) and should also overload display(d::Display, x)
+# to display x in whatever MIME type is preferred by the Display and
+# is writable by x. display(..., x) should throw a MethodError if x
+# cannot be displayed. The return value of display(...) is up to the
+# Display type.
+
+abstract Display
+
+# it is convenient to accept strings instead of ::MIME
+display(d::Display, mime::String, x) = display(d, MIME(mime), x)
+display(mime::String, x) = display(MIME(mime), x)
+displayable(d::Display, mime::String) = displayable(d, MIME(mime))
+displayable(mime::String) = displayable(MIME(mime))
+
+# simplest display, which only knows how to display text/plain
+immutable TextDisplay <: Display
+ io::IO
+end
+display(d::TextDisplay, ::@MIME("text/plain"), x) =
+ writemime(d.io, MIME("text/plain"), x)
+display(d::TextDisplay, x) = display(d, MIME("text/plain"), x)
+
+import Base: close, flush
+flush(d::TextDisplay) = flush(d.io)
+close(d::TextDisplay) = close(d.io)
+
+###########################################################################
+# We keep a stack of Displays, and calling display(x) uses the topmost
+# Display that is capable of displaying x (doesn't throw an error)
+
+const displays = Display[]
+function pushdisplay(d::Display)
+ global displays
+ push!(displays, d)
+end
+popdisplay() = pop!(displays)
+function popdisplay(d::Display)
+ for i = length(displays):-1:1
+ if d == displays[i]
+ return splice!(displays, i)
+ end
+ end
+ throw(KeyError(d))
+end
+function reinit_displays()
+ empty!(displays)
+ pushdisplay(TextDisplay(STDOUT))
+end
+
+function display(x)
+ for i = length(displays):-1:1
+ try
+ return display(displays[i], x)
+ catch e
+ if !isa(e, MethodError)
+ rethrow()
+ end
+ end
+ end
+ throw(MethodError(display, (x,)))
+end
+
+function display(m::MIME, x)
+ for i = length(displays):-1:1
+ try
+ return display(displays[i], m, x)
+ catch e
+ if !isa(e, MethodError)
+ rethrow()
+ end
+ end
+ end
+ throw(MethodError(display, (m, x)))
+end
+
+displayable{D<:Display,mime}(d::D, ::MIME{mime}) =
+ method_exists(display, (D, MIME{mime}, Any))
+
+function displayable(m::MIME)
+ for d in displays
+ if displayable(d, m)
+ return true
+ end
+ end
+ return false
+end
+
+###########################################################################
+# The redisplay method can be overridden by a Display in order to
+# update an existing display (instead of, for example, opening a new
+# window), and is used by the IJulia interface to defer display
+# until the next interactive prompt. This is especially useful
+# for Matlab/Pylab-like stateful plotting interfaces, where
+# a plot is created and then modified many times (xlabel, title, etc.).
+
+function redisplay(x)
+ for i = length(displays):-1:1
+ try
+ return redisplay(displays[i], x)
+ catch e
+ if !isa(e, MethodError)
+ rethrow()
+ end
+ end
+ end
+ throw(MethodError(redisplay, (x,)))
+end
+
+function redisplay(m::Union(MIME,String), x)
+ for i = length(displays):-1:1
+ try
+ return redisplay(displays[i], m, x)
+ catch e
+ if !isa(e, MethodError)
+ rethrow()
+ end
+ end
+ end
+ throw(MethodError(redisplay, (m, x)))
+end
+
+# default redisplay is simply to call display
+redisplay(d::Display, x) = display(d, x)
+redisplay(d::Display, m::Union(MIME,String), x) = display(d, m, x)
+
+###########################################################################
+
+end # module
View
1  base/stream.jl
@@ -232,6 +232,7 @@ function reinit_stdio()
global STDIN = init_stdio(ccall(:jl_stdin_stream ,Ptr{Void},()),0)
global STDOUT = init_stdio(ccall(:jl_stdout_stream,Ptr{Void},()),1)
global STDERR = init_stdio(ccall(:jl_stderr_stream,Ptr{Void},()),2)
+ reinit_displays() # since Multimedia.displays uses STDOUT as fallback
end
flush(::TTY) = nothing
View
4 base/sysimg.jl
@@ -62,6 +62,8 @@ include("utf8.jl")
include("iobuffer.jl")
include("string.jl")
include("regex.jl")
+include("base64.jl")
+importall .Base64
# system & environment
include("libc.jl")
@@ -80,6 +82,8 @@ include("stat.jl")
include("fs.jl")
importall .FS
include("process.jl")
+include("multimedia.jl")
+importall .Multimedia
reinit_stdio()
ccall(:jl_get_uv_hooks, Void, ())
include("grisu.jl")
View
12 doc/helpdb.jl
@@ -1827,6 +1827,18 @@
"),
+("Text I/O","Base","base64","base64(stream, args...)
+
+ Given a \"write\"-like function \"writefunc\", which takes an I/O
+ stream as its first argument, \"base64(writefunc, args...)\"
+ calls \"writefunc\" to write \"args...\" to a base64-encoded string,
+ and returns the string. \"base64(args...)\" is equivalent to
+ \"base64(write, args...)\": it converts its arguments into bytes
+ using the standard \"write\" functions and returns the base64-encoded
+ string.
+
+"),
+
("Memory-mapped I/O","Base","mmap_array","mmap_array(type, dims, stream[, offset])
Create an \"Array\" whose values are linked to a file, using
View
185 doc/stdlib/base.rst
@@ -1010,13 +1010,13 @@ I/O
.. function:: readbytes!(stream, b::Vector{Uint8}, nb=length(b))
- Read at most nb bytes from the stream into b, returning the
- number of bytes read (increasing the size of b as needed).
+ Read at most ``nb`` bytes from the stream into ``b``, returning the
+ number of bytes read (increasing the size of ``b`` as needed).
.. function:: readbytes(stream, nb=typemax(Int))
- Read at most nb bytes from the stream, returning a
- Vector{Uint8} of the bytes read.
+ Read at most ``nb`` bytes from the stream, returning a
+ ``Vector{Uint8}`` of the bytes read.
.. function:: position(s)
@@ -1186,6 +1186,183 @@ Text I/O
Equivalent to ``writedlm`` with ``delim`` set to comma.
+.. function:: Base64Pipe(ostream)
+
+ Returns a new write-only I/O stream, which converts any bytes written
+ to it into base64-encoded ASCII bytes written to ``ostream``. Calling
+ ``close`` on the ``Base64Pipe`` stream is necessary to complete the
+ encoding (but does not close ``ostream``).
+
+.. function:: base64(writefunc, args...)
+ base64(args...)
+
+ Given a ``write``-like function ``writefunc``, which takes an I/O
+ stream as its first argument, ``base64(writefunc, args...)``
+ calls ``writefunc`` to write ``args...`` to a base64-encoded string,
+ and returns the string. ``base64(args...)`` is equivalent to
+ ``base64(write, args...)``: it converts its arguments into bytes
+ using the standard ``write`` functions and returns the base64-encoded
+ string.
+
+Multimedia I/O
+--------------
+
+Just as text output is performed by ``print`` and user-defined types
+can indicate their textual representation by overloading ``show``,
+Julia provides a standardized mechanism for rich multimedia output
+(such as images, formatted text, or even audio and video), consisting
+of three parts:
+
+* A function ``display(x)`` to request the richest available multimedia
+ display of a Julia object ``x`` (with a plain-text fallback).
+* Overloading ``writemime`` allows one to indicate arbitrary multimedia
+ representations (keyed by standard MIME types) of user-defined types.
+* Multimedia-capable display backends may be registered by subclassing
+ a generic ``Display`` type and pushing them onto a stack of display
+ backends via ``pushdisplay``.
+
+The base Julia runtime provides only plain-text display, but richer
+displays may be enabled by loading external modules or by using graphical
+Julia environments (such as the IPython-based IJulia notebook).
+
+.. function:: display(x)
+ display(d::Display, x)
+ display(mime, x)
+ display(d::Display, mime, x)
+
+ Display ``x`` using the topmost applicable display in the display stack,
+ typically using the richest supported multimedia output for ``x``, with
+ plain-text ``STDOUT`` output as a fallback. The ``display(d, x)`` variant
+ attempts to display ``x`` on the given display ``d`` only, throwing
+ a ``MethodError`` if ``d`` cannot display objects of this type.
+
+ There are also two variants with a ``mime`` argument (a MIME type
+ string, such as ``"image/png"``) attempt to display ``x`` using the
+ requesed MIME type *only*, throwing a ``MethodError`` if this type
+ is not supported by either the display(s) or by ``x``. With these
+ variants, one can also supply the "raw" data in the requested MIME
+ type by passing ``x::String`` (for MIME types with text-based storage,
+ such as text/html or application/postscript) or ``x::Vector{Uint8}``
+ (for binary MIME types).
+
+.. function:: redisplay(x)
+ redisplay(d::Display, x)
+ redisplay(mime, x)
+ redisplay(d::Display, mime, x)
+
+ By default, the `redisplay` functions simply call ``display``. However,
+ some display backends may override ``redisplay`` to modify an existing
+ display of ``x`` (if any). Using ``redisplay`` is also a hint to the
+ backend that ``x`` may be redisplayed several times, and the backend
+ may choose to defer the display until (for example) the next interactive
+ prompt.
+
+.. function:: displayable(mime)
+ displayable(d::Display, mime)
+
+ Returns a boolean value indicating whether the given ``mime`` type (string)
+ is displayable by any of the displays in the current display stack, or
+ specifically by the display ``d`` in the second variant.
+
+.. function:: writemime(stream, mime, x)
+
+ The ``display`` functions ultimately call ``writemime`` in order to
+ write an object ``x`` as a given ``mime`` type to a given I/O
+ ``stream`` (usually a memory buffer), if possible. In order to
+ provide a rich multimedia representation of a user-defined type
+ ``T``, it is only necessary to define a new ``writemime`` method for
+ ``T``, via: ``writemime(stream, ::@MIME(mime), x::T) = ...``, where
+ ``mime`` is a MIME-type string and the function body calls
+ ``write`` (or similar) to write that representation of ``x`` to
+ ``stream``.
+
+ For example, if you define a ``MyImage`` type and know how to write
+ it to a PNG file, you could define a function ``writemime(stream,
+ ::@MIME("image/png"), x::MyImage) = ...``` to allow your images to
+ be displayed on any PNG-capable ``Display`` (such as IJulia).
+ As usual, be sure to ``import Base.writemime`` in order to add
+ new methods to the built-in Julia function ``writemime``.
+
+ Technically, the ``@MIME(mime)`` macro defines a singleton type for
+ the given ``mime`` string, which allows us to exploit Julia's
+ dispatch mechanisms in determining how to display objects of any
+ given type.
+
+.. function:: mimewritable(mime, T::Type)
+
+ Returns a boolean value indicating whether or not objects of type
+ ``T`` can be written as the given ``mime`` type. (By default, this
+ is determined automatically by the existence of the corresponding
+ ``writemime`` function.)
+
+.. function:: reprmime(mime, x)
+
+ Returns a ``String`` or ``Vector{Uint8}`` containing the
+ representation of ``x`` in the requested ``mime`` type, as written
+ by ``writemime`` (throwing a ``MethodError`` if no appropriate
+ ``writemime`` is available). A ``String`` is returned for MIME
+ types with textual representations (such as ``"text/html"`` or
+ ``"application/postscript"``), whereas binary data is returned as
+ ``Vector{Uint8}``. (The function ``istext(mime)`` returns whether
+ or not Julia treats a given ``mime`` type as text.)
+
+ As a special case, if ``x`` is a ``String`` (for textual MIME types)
+ or a ``Vector{Uint8}`` (for binary MIME types), the ``reprmime`` function
+ assumes that ``x`` is already in the requested ``mime`` format and
+ simply returns ``x``.
+
+.. function:: stringmime(mime, x)
+
+ Returns a ``String`` containing the representation of ``x`` in the
+ requested ``mime`` type. This is similar to ``reprmime`` except
+ that binary data is base64-encoded as an ASCII string.
+
+As mentioned above, one can also define new display backends. For
+example, a module that can display PNG images in a window can register
+this capability with Julia, so that calling `display(x)` on types
+with PNG representations will automatically display the image using
+the module's window.
+
+In order to define a new display backend, one should first create a
+subtype ``D`` of the abstract class ``Display``. Then, for each MIME
+type (``mime`` string) that can be displayed on ``D``, one should
+define a function ``display(d::D, ::@MIME(mime), x) = ...`` that
+displays ``x`` as that MIME type, usually by calling ``reprmime(mime,
+x)``. A ``MethodError`` should be thrown if ``x`` cannot be displayed
+as that MIME type; this is automatic if one calls ``reprmime``.
+Finally, one should define a function ``display(d::D, x)`` that
+queries ``mimewritable(mime, x)`` for the ``mime`` types supported by
+``D`` and displays the "best" one; a ``MethodError`` should be thrown
+if no supported MIME types are found for ``x``. Similarly, some
+subtypes may wish to override ``redisplay(d::D, ...)``. (Again, one
+should ``import Base.display`` to add new methods to ``display``.)
+The return values of these functions are up to the implementation
+(since in some cases it may be useful to return a display "handle" of
+some type). The display functions for ``D`` can then be called
+directly, but they can also be invoked automatically from
+``display(x)`` simply by pushing a new display onto the display-backend
+stack with:
+
+.. function:: pushdisplay(d::Display)
+
+ Pushes a new display ``d`` on top of the global display-backend
+ stack. Calling ``display(x)`` or ``display(mime, x)`` will display
+ ``x`` on the topmost compatible backend in the stack (i.e., the
+ topmost backend that does not throw a ``MethodError``).
+
+.. function:: popdisplay()
+ popdisplay(d::Display)
+
+ Pop the topmost backend off of the display-backend stack, or the
+ topmost copy of ``d`` in the second variant.
+
+.. function:: TextDisplay(stream)
+
+ Returns a ``TextDisplay <: Display``, which can display any object
+ as the text/plain MIME type (only), writing the text representation
+ to the given I/O stream. (The text representation is the same
+ as the way an object is printed in the Julia REPL.)
+
Memory-mapped I/O
-----------------
Something went wrong with that request. Please try again.