Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] MDX Support #1776

Open
sw-yx opened this issue Sep 26, 2018 · 35 comments
Open

[Feature Request] MDX Support #1776

sw-yx opened this issue Sep 26, 2018 · 35 comments

Comments

@sw-yx
Copy link

@sw-yx sw-yx commented Sep 26, 2018

See #2064 for technical requirements.

Is your feature request related to a problem? Please describe.

I want to use MDX with NetlifyCMS. But I can't.

Describe the solution you'd like

NetlifyCMS should:

  • detect my .mdx files
  • not attempt to WYSIWYG my mdx files, but it can render in the right pane

Describe alternatives you've considered

nil. vuepress was mentioned in another issue

Additional context

some background chat done between me and @erquhart on the netlify internal slack as well as cms channel: https://netlify.slack.com/archives/C1C30FQFM/p1537919621000100

so i converted the gatsby netlifycms template to use mdx: https://gatsby-starter-netlify-cms-with-mdx.netlify.com/
source: https://github.com/sw-yx/gatsby-starter-netlify-cms-1
however netlifycms just totally fails to recognize mdx files instead of md files. i think the problem comes in config.yml where it is just hardcoded to parse markdown
in particular this line https://github.com/sw-yx/gatsby-starter-netlify-cms-1/blob/master/static/admin/config.yml#L11
what do you folks think we should do here?

@verythorough:

The collection field accepts an extension sub-field, but mdx is not a valid value for that right now:
https://www.netlifycms.org/docs/configuration-options/#collections
If it was added, we'd also need to add an mdx parser of some kind. Markdown is currently parsed/generated with Slate & Remark

@erquhart:

However we do it, MDX will require a fair amount of effort within the CMS itself.
Your work made me realize mdx has it's own extension - if that's always the case, we may be able to tackle a lot of this at the format level, before the file content ever reaches the editor.
To wit, the CMS reads in files and parses them before storing them in state, so maybe mdx files get some extra state pointing to the components they import, and those are then made available to the editor, but the markdown string that's passed to the editor is stripped of those import/export statements.

@sw-yx
Copy link
Author

@sw-yx sw-yx commented Sep 26, 2018

one alternative i realized overnight is that we can write MDX files with an MD extension: https://gatsby-mdx.netlify.com/guides/mdx-file-extensions

so i did that to my own post: with these commits: https://github.com/sw-yx/gatsby-starter-netlify-cms-1/commit/07f6bd01d5f2133450c833deffba00a3107f9d99

and it worked! https://gatsby-starter-netlify-cms-with-mdx.netlify.com/blog/2017-01-04-this-is-an-mdx-post

Here's how it looks in netlify cms in markdown view

image

and the rich text view:

image

I can probably put in more work to fix the rendering on the right panel.

is this what we want? treat MDX as MD files? is that too hacky or do we want to have a special flow for MDX files?

Loading

@erquhart erquhart removed their assignment Sep 26, 2018
@erquhart
Copy link
Contributor

@erquhart erquhart commented Sep 26, 2018

Hmm yeah I guess if the editor doesn't know about MDX it won't mess with any of it (already treats xml style markup as html and ignores). Really cool that MDX isn't 100% blocked for folks that really want it.

To announce support, however, we'll need preview support in place.

Loading

@tech4him1
Copy link
Contributor

@tech4him1 tech4him1 commented Sep 28, 2018

one alternative i realized overnight is that we can write MDX files with an MD extension

@sw-yx
Technically, you should be able to set on the collection:

extension: mdx
format: frontmatter

You can then leave the files with an mdx extension, and they will be treated like a regular MD file.

The collection field accepts an extension sub-field, but mdx is not a valid value for that right now:
https://www.netlifycms.org/docs/configuration-options/#collections

@verythorough That's not exactly correct -- the extension field will take any value (that's how people can use it for en.md, for example). The only difference if it is not one of those listed values is that the format must be set as well, since it cannot be inferred. I'm thinking we should update the linked documentation to make that more clear. Thoughts?

Loading

@verythorough
Copy link
Contributor

@verythorough verythorough commented Sep 28, 2018

@tech4him1 Right. The docs do actually say that in the two paragraphs that follow, but people (like me! and I wrote it!) might not read that far.

My Friday-afternoon brain isn't coming up with a clearer wording. Maybe we should file a separate issue for it. I'll drop it on my to-do list to look at later.

Another semi-related thing I think I'll make a feature request for is a "raw" format - something that does no parsing or previewing, but passes through the raw text, allows editing, and saves as typed. It's obviously not full support, but could be a nice option for allowing the authentication/browsing/saving flow of the CMS for formats that aren't supported (like rST, for example).

Loading

@sw-yx
Copy link
Author

@sw-yx sw-yx commented Oct 1, 2018

yes. my kind of ideal (and i know i dont speak for everyone) is the dev.to workflow - where there is ONLY "raw" and then the preview pane. this way it's fast (because no wysiwyg) and extensible (mdx is a cinch). sorry i know this is classic scope creep :)

Loading

@sw-yx
Copy link
Author

@sw-yx sw-yx commented Dec 16, 2018

inserting ok-mdx is an option https://github.com/jxnblk/ok-mdx

Loading

@talves
Copy link
Collaborator

@talves talves commented Dec 16, 2018

@sw-yx is the only issue we have so far is to fix the preview (right pane)? or do you want full support from the editor also?

Loading

@sw-yx
Copy link
Author

@sw-yx sw-yx commented Dec 16, 2018

yup, preview is fine.basically a dev.to workflow would be seamless to add this. but also there’s probably config stuff we need to work out.

Loading

@erquhart
Copy link
Contributor

@erquhart erquhart commented Dec 17, 2018

The big issue is MDX import and export statements.

ok-mdx looks awesome, great find @sw-yx! It may face similar issues as codemirror getting integrated into slate, but if nothing else it's a good starting point.

Another note for folks following this: MDX is now under the Unified umbrella along with Remark and Rehype, the libraries our markdown parsing is already driven by. We should be able to support MDX without wrapping in a purpose built editor, but hard to say for sure until someone digs in.

Loading

@talves
Copy link
Collaborator

@talves talves commented Dec 17, 2018

I dug into it this weekend and rendering MDX during runtime for the preview is going to be a big ask.

Loading

@erquhart
Copy link
Contributor

@erquhart erquhart commented Dec 17, 2018

I expect it will look like this:

  • pluck out import/exports before parsing the file, and prepare references to the imported/exported components
  • add serialization logic to Remark for recognizing MDX imported React components
  • I think that's all

From there, the markdown-turned-html and React components can all be static rendered via react-dom and live updated with any changes (which is what we already do). The question is where to register the React components that the MDX is importing. Probably really simple to solve, but I haven't spent any time considering it yet.

Loading

@karolis-sh
Copy link
Contributor

@karolis-sh karolis-sh commented Dec 26, 2018

I'm building an MDX based CMS at work, tried a few things and as @erquhart mentioned one of the main issues for me was the imports.
I made a small wrapper mdx-scoped-runtime around @mdx-js/runtime package that strips down the imports and layout export syntax using babel standalone. The gotcha is that you have to provide the possible importable components via scope prop for the pseudo-dynamic imports to work.

Also made a widget with said runtime. It's limited with the pre-defined scope prop, but the preview seems to work as expected.

I'd like to expand the mdx-scoped-runtime with ability to analyse the imports and provide some validation maybe, but the gatsby cms build has some issues (gatsbyjs/gatsby#10326) with certain babel plugins.

EDIT: I actually refactored the solution, turns out you can remove the imports via remark plugins quite easily, so no @babel/standalone required.

Loading

@damassi
Copy link

@damassi damassi commented Jan 23, 2019

@buz-zard - the widget is working well 👌 It would be nice to find a way to make it work with gatsby-mdx (different package) global scope ChristopherBiscardi/gatsby-mdx#226, but for the time being the issue can be worked around by duplication.

Loading

@karolis-sh
Copy link
Contributor

@karolis-sh karolis-sh commented Jan 23, 2019

@damassi the mdx widget only uses the mdx-scoped-runtime. The CMS and gatsby pages are like 2 separate apps, so it really doesn't even matter what you use for gatsby+mdx handling.

Check the scoped runtime demo - it's just a minimal React hello-world app.

As for the global scope, check the readme - it's passed via scope prop.

I should probably extend the readme a bit 🤔

Loading

@damassi
Copy link

@damassi damassi commented Jan 26, 2019

For those who are curious I have a full working starter example here , and here's the cms.js file.

Loading

@Undistraction
Copy link
Contributor

@Undistraction Undistraction commented Jan 26, 2019

@damassi Thanks. This looks excellent. Do all the gatsby-remark ... plugins work with mdx content? Other than the rendering can it be used interchangeably with plain markdown docs.

Loading

@damassi
Copy link

@damassi damassi commented Jan 26, 2019

HI @Undistraction - You can pass in remark plugins at the gatsby-config.js level via gatsby-mdx. And yup, gatsby-mdx supports .md extensions as well.

Unrelated, it's important to note that when working with React components in .mdx and making edits in NetlifyCMS the globalScope key located just above does not apply to the scope required for NetlifyCMS: whatever is passed into one has to be passed into the other as well, e.g.,

// gatsby-config.js 

globalScope: `
  import *  as Components from 'rebass'
  export default { 
    ...Components 
  }
`
// cms.js
import * as Components from 'rebass'
...
scope: Components 

Loading

@Undistraction
Copy link
Contributor

@Undistraction Undistraction commented Jan 26, 2019

@damassi Thanks. This is helpful.

Loading

@talves
Copy link
Collaborator

@talves talves commented Jan 26, 2019

@damassi tested the Button import out on the build and works now. Again, thanks.

I wonder how easy it would be to take this to a CRA solution.

Loading

@erquhart erquhart added this to Ready For Development in Priority Issues Feb 13, 2019
@SachaG
Copy link

@SachaG SachaG commented Mar 23, 2019

Trying to use gatsby-mdx, I'm getting an error that I think is related when doing gatsby develop:

 ERROR  Failed to compile with 1 errors                                                                                                               12:09:38

 error  in ./node_modules/gatsby-mdx/wrap-root-element.js

Module parse failed: Unexpected token (38:11)
You may need an appropriate loader to handle this file type.
|       ({ guard }) => (guard ? guard(props) : true)
|     );
>     return <Component {...props} />;
|   };
|

 @ ./node_modules/gatsby-mdx/gatsby-browser.js 1:0-39 2:31-35
 @ ./.cache/api-runner-browser-plugins.js
 @ ./.cache/api-runner-browser.js
 @ ./.cache/page-renderer.js
 @ ./.cache/json-store.js
 @ ./.cache/public-page-renderer-dev.js
 @ ./.cache/public-page-renderer.js
 @ ./.cache/gatsby-browser-entry.js
 @ ./src/components/Layout.js
 @ ./src/templates/about-page.js
 @ ./src/cms/preview-templates/AboutPagePreview.js
 @ ./src/cms/cms.js
 @ multi ./node_modules/gatsby-plugin-netlify-cms/cms.js ./src/cms/cms.js ./node_modules/gatsby-plugin-netlify-cms/cms-identity.js

Somehow wrap-root-element.js is not getting treated as jsx? Could this be related to NetlifyCMS or am I off the mark?

My gatsby info config:


  System:
    OS: macOS 10.14.3
    CPU: (8) x64 Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 8.12.0 - ~/.nvm/versions/node/v8.12.0/bin/node
    Yarn: 1.13.0 - /usr/local/bin/yarn
    npm: 6.7.0 - ~/.nvm/versions/node/v8.12.0/bin/npm
  Languages:
    Python: 3.6.4 - /usr/local/bin/python
  Browsers:
    Chrome: 73.0.3683.86
    Firefox: 65.0.1
    Safari: 12.0.3
  npmPackages:
    gatsby: ^2.1.37 => 2.1.37
    gatsby-image: ^2.0.34 => 2.0.34
    gatsby-mdx: ^0.4.4 => 0.4.4
    gatsby-plugin-netlify: ^2.0.13 => 2.0.13
    gatsby-plugin-netlify-cms: ^3.0.16 => 3.0.16
    gatsby-plugin-purgecss: ^3.1.0 => 3.1.0
    gatsby-plugin-react-helmet: ^3.0.10 => 3.0.10
    gatsby-plugin-sass: ^2.0.11 => 2.0.11
    gatsby-plugin-sharp: ^2.0.29 => 2.0.29
    gatsby-remark-copy-linked-files: ^2.0.11 => 2.0.11
    gatsby-remark-images: ^3.0.10 => 3.0.10
    gatsby-remark-relative-images: ^0.2.2 => 0.2.2
    gatsby-source-filesystem: ^2.0.27 => 2.0.27
    gatsby-source-graphql: ^2.0.15 => 2.0.15
    gatsby-transformer-remark: ^2.3.4 => 2.3.4
    gatsby-transformer-sharp: ^2.1.17 => 2.1.17
  npmGlobalPackages:
    gatsby-cli: 2.4.17

Loading

@erquhart
Copy link
Contributor

@erquhart erquhart commented Apr 2, 2019

@SachaG it isn't clear to me how that would be caused by Netlify CMS - have you raised an issue in the Gatsby repo?

Loading

@SachaG
Copy link

@SachaG SachaG commented Apr 4, 2019

Not yet, I ended up putting MDX aside for now. But yeah, if nobody else is running into this then it's probably due to my Gatsby config.

Loading

@stale
Copy link

@stale stale bot commented Oct 29, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Loading

@mtliendo
Copy link

@mtliendo mtliendo commented Mar 29, 2020

Really loving NetlifyCMS and would like to see this integrated.

If possible, can we get a list of blockers so that the community can try and work towards solving them?

Also, Is there an opportunity to collaborate with the folks over at contentful on this? It seems like they've had success in getting this implemented.

Loading

@erezrokah
Copy link
Collaborator

@erezrokah erezrokah commented Mar 29, 2020

Hi @mtliendo, in the past months we've been trying to tackle the most upvoted features first:
https://github.com/netlify/netlify-cms/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc

Not sure whats is the definition for a blocker is in regards with new features.

Regrading collaboration, we would first need someone to take ownership of the issue and start working on it.

Loading

@mtliendo
Copy link

@mtliendo mtliendo commented Mar 29, 2020

Hi @erezrokah 👋 Thanks for the link, and it looks like github projects are being used, so that gave me some awesome insight into where this is at. Regarding blockers, the issue seems like it's gone stale after trying to work out the preview screen and general error handling, but having specifics would help others pick up where @sw-yx left off.

Sorry, I'm new to the repo, are issues open to anyone in the community to own or are they championed by someone on the Netlify team?

Loading

@erezrokah
Copy link
Collaborator

@erezrokah erezrokah commented Mar 29, 2020

Any issue is open for the community to contribute regardless of upvotes or prioritisation and will be backed by the Netlify team as much as possible :)
We mark some issues as good first issue, other ones might need some communication and discussion first.
Regardless of the issue it is always best to communicate first so we can assign an owner so people know the issue is taken.

If you'd like to work on this one let me know and I'll assign you.

Loading

@mtliendo
Copy link

@mtliendo mtliendo commented Mar 30, 2020

Sorry didn't mean to ghost you 😅. I've been in the CMS all weekend and would like to take on some tasks, this one might be a bit more involved for my first one though. I'll keep an eye on it and others in the meantime and ping back when I spot something.

Thanks for all the info and direction!

Loading

@awilderink
Copy link

@awilderink awilderink commented Apr 13, 2020

I got it working actually pretty nice together with React-Ace. Here my cms.jsx:

import React, { Component } from 'react'
import CMS from 'netlify-cms-app'
import IndexPagePreview from './preview-templates/IndexPagePreview'
import { components } from '../Theme'
import cloudinary from 'netlify-cms-media-library-cloudinary'
import withEmotion from './cssInjector'
import MDX from 'mdx-scoped-runtime'
import AceEditor from 'react-ace'
import 'ace-builds/src-noconflict/mode-jsx'
import 'ace-builds/src-noconflict/mode-html'
import 'ace-builds/src-noconflict/theme-monokai'

const isClient = typeof window !== 'undefined'
const isDevelopment = process.env.NODE_ENV === 'development'

if (isClient) {
  window.CMS_MANUAL_INIT = true
}

if (isDevelopment) {
  window.CMS_ENV = 'localhost_development'
}

class CodeEditor extends Component {
  render() {
    return (
      <AceEditor
        tabSize={2}
        mode="html" // or use jsx 
        theme="monokai"
        width="100%"
        value={this.props.value}
        onChange={this.props.onChange}
      />
    )
  }
}

CMS.registerPreviewTemplate('index', withEmotion(IndexPagePreview))

CMS.registerWidget(
  'mdx',
  CodeEditor,
  withEmotion(({ value }) => (
    <div className="netlify-cms-widget-mdx-preview">
      <MDX scope={components}>{value}</MDX>
    </div>
  ))
)

CMS.registerMediaLibrary(cloudinary)

CMS.init()

Loading

@narthur
Copy link

@narthur narthur commented Apr 22, 2020

I would love support for MDX to be built in to the editor as well as the preview, since I'm considering using MDX on a project where non-technical folks need to feel comfortable editing pages.

Loading

@erezrokah
Copy link
Collaborator

@erezrokah erezrokah commented Apr 23, 2020

@narthur, best way to push this it to upvote the issue since we use it for prioritisation: https://github.com/netlify/netlify-cms/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc

Loading

@nkuehn
Copy link

@nkuehn nkuehn commented Jun 18, 2020

@awilderink thanks for sharing your approach, looks interesting! Can you explain how your "withEmotion" helper works (we're using styled components for all the MD(x) elements) and whether it would work with the standard editor instead of Ace, too? Any tips appreciated!

Loading

@awilderink
Copy link

@awilderink awilderink commented Jul 9, 2020

@nkuehn Sorry for my late respons. The WithEmotion helper just inserts the prerendered CSS into the preview window. Here is the code:

import React from 'react'
import { CacheProvider } from '@emotion/core'
import createCache from '@emotion/cache'
import weakMemoize from '@emotion/weak-memoize'
import { ThemeProvider, Styled } from 'theme-ui'
import theme from '../gatsby-plugin-theme-ui'

const memoizedCreateCacheWithContainer = weakMemoize(container => {
  const newCache = createCache({ container })
  return newCache
})

export default Component => props => {
  const iframe = document.querySelector('#nc-root iframe')
  const iframeHeadElem = iframe && iframe.contentDocument.head

  if (!iframeHeadElem) {
    return null
  }

  return (
    <CacheProvider value={memoizedCreateCacheWithContainer(iframeHeadElem)}>
      <ThemeProvider theme={theme}>
        <Styled.root>
          <Component {...props} />
        </Styled.root>
      </ThemeProvider>
    </CacheProvider>
  )
}

I got it from somewhere else, but there are similar examples for styled-components too. I found a few examples here: #793

Loading

@karlhorky
Copy link

@karlhorky karlhorky commented May 22, 2021

Also, Is there an opportunity to collaborate with the folks over at contentful on this? It seems like they've had success in getting this implemented.

@mtliendo seems like the code behind this example is actually not all that complicated:

https://github.com/ChristopherBiscardi/gatsby-mdx/blob/master/ui-extensions/contentful/index.js

So probably it wouldn't even need a collaboration - just someone to actually take the time to do the work here.

Loading

@karlhorky
Copy link

@karlhorky karlhorky commented May 22, 2021

mdx-bundler by Kent C. Dodds may also be interesting here (uses xdm and esbuild for fast, powerful bundling): https://github.com/kentcdodds/mdx-bundler/

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Priority Issues
Needs Planning
Linked pull requests

Successfully merging a pull request may close this issue.

None yet