Allow spaces in names of files attached to markdown cells

[ianhi]

References

partially fixes: #8067:
This will fix the dragging and dropping behavior, but as noted #8062 (comment) the other issue raised there of typing in the name will remain as that is not supported by markdown.
partially addresses: #8062:
the spaces in names also came up there.

Code changes

call encodeURI when setting the attachment to a markdown cell. This allows the attachment to be found when the markdown is being rendered. I was able to confirm that this fixes the issues for nativeDrop and paste events. However, I don't know how to trigger the lm-drop events so I wasn't able to directly verify that this fixes the issue for those events. If these are meant to be triggered by dragging a file from the jupyterlab filebrowser then I was unable to do that and have a new issue to report.

User-facing changes

An image with spaces in the name that is drag and dropped or copied into a markdown cell will now render:

Backwards-incompatible changes

N/A

[jupyterlab-dev-mode]

Thanks for making a pull request to JupyterLab!

To try out this branch on binder, follow this link:

[jasongrout]

I put this in the 2.1 milestone, assuming we can finish it up by this weekend.

[ianhi]

Sounds good. The one other thing maybe worth considering is what the expected behavior is for non-image files? The ![]() notation which is always used is only valid for images. So if you drag a markdown file into a markdown cell it will paste the contents of the file and then add: ![markdown_file.md](attachment:markddown_file.md) which doesn't render.

[jasongrout]

The one other thing maybe worth considering is what the expected behavior is for non-image files?

Great question. Is it easy to determine if the file is an image? Do we have the mimetype in the clipboard data? I suppose we can guess based on extension as a last resort?

[jasongrout]

Reading up on the commonmark spec, apparently you can have a space in the filename, you just have surround the link with <>: https://spec.commonmark.org/0.29/#example-486

[jasongrout]

To quote the commonmark spec: https://spec.commonmark.org/0.29/#links

A link destination consists of ... a sequence of zero or more characters between an opening < and a closing > that contains no line breaks or unescaped < or > characters,

[ianhi]

Should be possible to determine if a file is an image. The clipboardData has a .type attribute that should be confined to one of these "Mandatory" data types (https://w3c.github.io/clipboard-apis/#reading-from-clipboard):

text/plain

text/uri-list

text/csv

text/css

text/html

application/xhtml+xml

image/png

image/jpg, image/jpeg

image/gif

image/svg+xml

application/xml, text/xml

application/javascript

application/json

application/octet-stream

Seems to me that the most reasonable to support are:

text/plain
text/csv
image/*

[jasongrout]

So what if we just take the attachmentName, do a manual regex for newlines and < and > to replace those with the HTML entities, and enclose the link in pointy brackets? It should be much more readable then. We can even determine if we should put pointy brackets around it based on if it has a space if we want the resulting markdown to look nicer.

[ianhi]

When I just type in a reference to a file using < and > doesn't fix the issues with spaces for me.
Renders: ![image](<image.png>)
Doesn't render: ![with spaces](<name with space.png>)
Renders: ![name with space.png](<name%20with%20space.png>)

maybe that's an issue with markedjs?

[jasongrout]

maybe that's an issue with markedjs?

Yes, it seems so

[ianhi]

Also, while the types are well defined for clipboardEvents they are not defined for drop events:
https://html.spec.whatwg.org/multipage/dnd.html#the-drag-data-item-type-string

The drag data item type string
A Unicode string giving the type or format of the data, generally given by a MIME type. Some values that are not MIME types are special-cased for legacy reasons. The API does not enforce the use of MIME types; other values can be used as well. In all cases, however, the values are all converted to ASCII lowercase by the API.

[jasongrout]

It does seem that markdown-it supports the <> style links: demo

[jasongrout]

c.f. #272

[ianhi]

So is the conclusion to use this solution until markedjs implements < or until jupyterlab switches to something like markdown-it?

1

Video could also be supported using a video tag. For example:

diff --git a/packages/cells/src/widget.ts b/packages/cells/src/widget.ts
index c013e1ffe..94761638c 100644
--- a/packages/cells/src/widget.ts
+++ b/packages/cells/src/widget.ts
@@ -1327,8 +1327,14 @@ export abstract class AttachmentsCell extends Cell {
       const encodedData = matches[3];
       const bundle: nbformat.IMimeBundle = { [mimeType]: encodedData };
       const URI = encodeURI(blob.name)
-      this.model.attachments.set(URI, bundle);
-      this.updateCellSourceWithAttachment(blob.name, URI);
+      if (mimeType.includes('image')){
+        this.model.attachments.set(URI, bundle);
+        this.updateCellSourceWithAttachment(blob.name, URI);
+      } else if(mimeType.includes('video')){
+        this.model.attachments.set(URI, bundle);
+        const textToBeAppended = `<video controls src='attachment:${URI}'></video>`
+        this.model.value.insert(this.model.value.text.length, textToBeAppended);
+      }
     };
     reader.onerror = evt => {
       console.error(`Failed to attach ${blob.name}` + evt);

although I'm struggling with attachment:video.mp4 not being a valid URI for the video tag. Though the equivalent works fine for img tags.

2

Because attachments.set is called without checking whether that URI is already being used you can end up overwriting already embedded images. This is primarily an issue when pasting a screenshot, as the paste is always named image.png. So it's currently impossible to paste two distinct images into the same markdown cell. Does a fix for that belong in this PR, or perhaps a different one?

[jasongrout]

So is the conclusion to use this solution until markedjs implements < or until jupyterlab switches to something like markdown-it?

Yes, I think so.

Video could also be supported using a video tag.

Awesome!

although I'm struggling with attachment:video.mp4 not being a valid URI for the video tag. Though the equivalent works fine for img tags.

This is because the attachment resolver, what actually converts this attachment: syntax to a data url for the html on the page, filters for only images:

jupyterlab/packages/attachments/src/model.ts

Lines 412 to 419 in c4a0f4a

	if (
	mimeType === undefined ||
	imageRendererFactory.mimeTypes.indexOf(mimeType) === -1
	) {
	return Promise.reject(
	`Cannot render unknown image mime type "${mimeType}".`
	);
	}

[jasongrout]

A few more review comments.

Another way to approach the attachment conflicts is to generate a UUID, which becomes the URI, so you have unique UUIDs for every attachment. The bad thing about a UUID is that there is no intrinsic meaning in the names of entries in the attachments list, but maybe that's okay? The association to a file name is in the markdown text if a person wants to cross reference.

[ianhi]

This is because the attachment resolver, what actually converts this attachment: syntax to a data url for the html on the page, filters for only images:

Yeah. When I modify that line I can easily embed videos by dragging them in. The question is then where to keep the list of valid mimetypes. It seems to me that both AttachmentsCell and AttachmentsResolver ought to share a ReadonlyArray of their supported mimetypes. This would also help solve the issues of spurious embeddings when you drag over a non image or video filetype. I'm pretty lost on what the correct place to put such a thing is suggestions welcome on that.

Currently I've changed the AttachmentsResolver like so:

diff --git a/packages/attachments/src/model.ts b/packages/attachments/src/model.ts
index 973e4d0d6..3dacd3224 100644
--- a/packages/attachments/src/model.ts
+++ b/packages/attachments/src/model.ts
@@ -14,7 +14,6 @@ import {
 import {
   IAttachmentModel,
   AttachmentModel,
-  imageRendererFactory
 } from '@jupyterlab/rendermime';
 
 import { IRenderMime } from '@jupyterlab/rendermime-interfaces';
@@ -378,6 +377,7 @@ export class AttachmentsResolver implements IRenderMime.IResolver {
   constructor(options: AttachmentsResolver.IOptions) {
     this._parent = options.parent || null;
     this._model = options.model;
+    this.supportedTypes = ['video/mp4','video/webm','video/ogg','image/bmp', 'image/png', 'image/jpeg', 'image/gif']; 
   }
   /**
    * Resolve a relative url to a correct server path.
@@ -411,7 +411,7 @@ export class AttachmentsResolver implements IRenderMime.IResolver {
     // Only support known safe types:
     if (
       mimeType === undefined ||
-      imageRendererFactory.mimeTypes.indexOf(mimeType) === -1
+      this.supportedTypes.indexOf(mimeType) === -1
     ) {
       return Promise.reject(
         `Cannot render unknown image mime type "${mimeType}".`
@@ -434,6 +434,7 @@ export class AttachmentsResolver implements IRenderMime.IResolver {
 
   private _model: IAttachmentsModel;
   private _parent: IRenderMime.IResolver | null;
+  readonly supportedTypes: ReadonlyArray<string>;
 }
 
 /**

[ianhi]

The uuid approach also removes the need to encode URIs, and helps negate possibility someone seeing the filename in the embedding and thinking that if they change the file on disk then the markdown cell will update.

[jasongrout]

The uuid approach also removes the need to encode URIs, and helps negate possibility someone seeing the filename in the embedding and thinking that if they change the file on disk then the markdown cell will update.

Oh yeah, I hadn't thought of that. Switching to UUID makes it clearer that this is a snapshot of the file, and it is different than actually linking to the file on disk. Yeah, I would suggest moving to UUID for that reason, but maybe keep the extension?

[ianhi]

I switched to using UUIDs and keep the extension when one exists. I also check if the file is an image type because that's the only file type supported by the ![]() syntax.

If possible it'd be nice to add support for videos as well. Though as noted, that would require changes to the attachments package. If these are only used for attaching to cells this seems pretty easy. But if they are used elsewhere I'm not sure how best to change them to allow the attachment resolver to resolve videos in addition to images. My concern is that it might require adding a RenderedVideo class to https://github.com/jupyterlab/jupyterlab/blob/master/packages/rendermime/src/widgets.ts ?

[jasongrout]

I like the new UUID support! I have a few more suggestions for it. I think we are converging to something really nice here!

[jasongrout]

Below, we now check to see if the data is an image mimetype. Should we do the same here?

[ianhi]

There's already some mimetype filtering happening in this function when if makes the array of supportedMimetypes

jupyterlab/packages/cells/src/widget.ts

Lines 1245 to 1247 in 5977421

	const supportedMimeTypes = toArray(
	filter(event.mimeData.types(), mimeType => {
	if (mimeType === CONTENTS_MIME_RICH) {

I hoped that this was doing enough filtering. I wasn't able to test this though, because I don't seem to be able to trigger this event. Should this be triggerable by dragging an image from the jupyterlab filebrowser?

[jasongrout]

If possible it'd be nice to add support for videos as well. Though as noted, that would require changes to the attachments package. If these are only used for attaching to cells this seems pretty easy. But if they are used elsewhere I'm not sure how best to change them to allow the attachment resolver to resolve videos in addition to images. My concern is that it might require adding a RenderedVideo class to https://github.com/jupyterlab/jupyterlab/blob/master/packages/rendermime/src/widgets.ts ?

I don't think it requires anything added to the rendermime package - that's completely separate from what we are doing here. I think how it would work is that we construct an HTML video element with a url of attachment:..., and then the attachment resolver needs to just convert the video to a data URI. This line in the resolver:

jupyterlab/packages/attachments/src/model.ts

Lines 411 to 419 in c4a0f4a

	// Only support known safe types:
	if (
	mimeType === undefined ||
	imageRendererFactory.mimeTypes.indexOf(mimeType) === -1
	) {
	return Promise.reject(
	`Cannot render unknown image mime type "${mimeType}".`
	);
	}

I think just uses the rendermime to check to see if it is a "safe" data format. I think it's probably a pretty poor check, actually. Perhaps instead we could whitelist some "safe" formats like image/png as well as some video ones, and not refer to the rendermime?

On the other hand, supporting video would be a nice place to draw the line for this PR, and put in a new PR to support that.

[ianhi]

Perhaps instead we could whitelist some "safe" formats like image/png as well as some video ones, and not refer to the rendermime?

Yeah. I would like to add a readonly array supportedTypes to both attachmentsResolver and attachments. My one worry is that this might mess with extensions that also attach data?

On the other hand, supporting video would be a nice place to draw the line for this PR, and put in a new PR to support that.

That seems reasonable. A side effect of this PR is that I'm now working towards adding a videoRenderer, so a separate PR could include a mimerenderer and support for video and audio in markdown cells.

[saulshanabrook]

Looks good overall!

Just one little typo.