New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ghost Importer #950

Closed
ErisDS opened this Issue Sep 30, 2013 · 27 comments

Comments

Projects
None yet
10 participants
@ErisDS
Member

ErisDS commented Sep 30, 2013

Further from #854

We want to write a Ghost app which:

  1. Import from the WordPress WXR format. This should work for WordPress and Tumblr on account of tumblr2wordpress.benapps.net/‎
  2. Imports from Blogger

both of these are XML formats, which we should be able to process using xml2js.

Just need to finish investigating / completing the plugin API stuff in #854

The repo is here: https://github.com/TryGhost/Ghost-Importer

@JohnONolan

This comment has been minimized.

Show comment
Hide comment
@JohnONolan

JohnONolan Oct 1, 2013

Member

It should be a single "Ghost Importer" plugin which supports multiple sources

Member

JohnONolan commented Oct 1, 2013

It should be a single "Ghost Importer" plugin which supports multiple sources

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 1, 2013

Member

That seems like a bad idea to me. There's no way we'll ever cover every option, so other people will want to build our own. That means we'll end up with our one being called 'Ghost Importer' and then someone else building 'Ghost Medium Importer' etc.. seems weird. People will only ever want to import from one source, so they'll only ever need to download one of them? It's not one of those things where having many will result in people having 10 extra plugins.

Member

ErisDS commented Oct 1, 2013

That seems like a bad idea to me. There's no way we'll ever cover every option, so other people will want to build our own. That means we'll end up with our one being called 'Ghost Importer' and then someone else building 'Ghost Medium Importer' etc.. seems weird. People will only ever want to import from one source, so they'll only ever need to download one of them? It's not one of those things where having many will result in people having 10 extra plugins.

@javorszky

This comment has been minimized.

Show comment
Hide comment
@javorszky

javorszky Oct 1, 2013

Member

Can plugins extend plugins? With WordPress I can make a separate plugin that

  • only works if, say, Gravity Forms is installed
  • and the only work of my plugin is to enhance Gravity Forms with some functionality they didn't have

I think as long as the Ghost Importer plugin has its own API, extending it with plugins isn't that a bad idea

Member

javorszky commented Oct 1, 2013

Can plugins extend plugins? With WordPress I can make a separate plugin that

  • only works if, say, Gravity Forms is installed
  • and the only work of my plugin is to enhance Gravity Forms with some functionality they didn't have

I think as long as the Ghost Importer plugin has its own API, extending it with plugins isn't that a bad idea

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 1, 2013

Member

In other contexts that might make sense, but I'm not sure that there will be anything in an importer plugin you'd want to extend. Guess it depends on what gets built and whether there is a shared toolset for parsing and manipulating data.

Member

ErisDS commented Oct 1, 2013

In other contexts that might make sense, but I'm not sure that there will be anything in an importer plugin you'd want to extend. Guess it depends on what gets built and whether there is a shared toolset for parsing and manipulating data.

@javorszky

This comment has been minimized.

Show comment
Hide comment
@javorszky

javorszky Oct 1, 2013

Member

The way the importer works in my mind is you hurl a file at it as input. That input is then parsed into something Ghost can understand, and then that data is thrown at Ghost.

The bit where the Ghost-ready data is thrown at Ghost is common. Parsing different inputs can be plugin-specific, and I don't think we'd need 7 plugins that all implement the "throw this at Ghost" functionality. Especially if 3 of them aren't updated anymore, so you end up importing from tumblr successfully, and then importing from movabletype fails / kills your database because the maintainer hadn't updated that in 4 months, meanwhile the schema changed ever so slightly on the Ghost end.

Member

javorszky commented Oct 1, 2013

The way the importer works in my mind is you hurl a file at it as input. That input is then parsed into something Ghost can understand, and then that data is thrown at Ghost.

The bit where the Ghost-ready data is thrown at Ghost is common. Parsing different inputs can be plugin-specific, and I don't think we'd need 7 plugins that all implement the "throw this at Ghost" functionality. Especially if 3 of them aren't updated anymore, so you end up importing from tumblr successfully, and then importing from movabletype fails / kills your database because the maintainer hadn't updated that in 4 months, meanwhile the schema changed ever so slightly on the Ghost end.

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 1, 2013

Member

This is OSS, what you describe will happen - because even if we make a 'Ghost Importer' plugin, other people won't necessarily extend it or add to it to make their own. They'll do their own thing and it will inevitably break if they don't keep it up to date.

You can see my first stabs at an importer plugin and how it would hook into Ghost here: ErisDS@5a14d35

I'm very interested in suggestions for improving it.

Member

ErisDS commented Oct 1, 2013

This is OSS, what you describe will happen - because even if we make a 'Ghost Importer' plugin, other people won't necessarily extend it or add to it to make their own. They'll do their own thing and it will inevitably break if they don't keep it up to date.

You can see my first stabs at an importer plugin and how it would hook into Ghost here: ErisDS@5a14d35

I'm very interested in suggestions for improving it.

@JohnONolan

This comment has been minimized.

Show comment
Hide comment
@JohnONolan

JohnONolan Oct 1, 2013

Member

There simply aren't that many platforms to import from - it doesn't make sense to split them into multiple confusing plugins when one will do.

  • WordPress
  • Blogger
  • Tumblr
  • Medium
  • Svbtle
  • Movable Type
  • Custom (some format which we specify)
  • RSS

Anything outside those arguable should be a separate plugin anyway

Member

JohnONolan commented Oct 1, 2013

There simply aren't that many platforms to import from - it doesn't make sense to split them into multiple confusing plugins when one will do.

  • WordPress
  • Blogger
  • Tumblr
  • Medium
  • Svbtle
  • Movable Type
  • Custom (some format which we specify)
  • RSS

Anything outside those arguable should be a separate plugin anyway

@jamesbloomer

This comment has been minimized.

Show comment
Hide comment
@jamesbloomer

jamesbloomer Oct 2, 2013

Member

However it's packaged, as one plugin or many, as @javorszky says there's the common bit, taking data in a standard format and importing it (that's what exists right?) and then there's the different bits converting other formats to the standard import interface. So practically there will be a module for each of the different blogs to import from and the common module. The work on the conversion modules could start as the format to convert from and to is known.

Member

jamesbloomer commented Oct 2, 2013

However it's packaged, as one plugin or many, as @javorszky says there's the common bit, taking data in a standard format and importing it (that's what exists right?) and then there's the different bits converting other formats to the standard import interface. So practically there will be a module for each of the different blogs to import from and the common module. The work on the conversion modules could start as the format to convert from and to is known.

@jgable

This comment has been minimized.

Show comment
Hide comment
@jgable

jgable Oct 28, 2013

Member

Seems like the import plugin stuff is best discussed in this thread. I took a lot at your importer plugin and I think it will work. We'll need to get my other PR in for the asynchronous doFilter or it won't work when the import actually has to save things to a database.

Is there a UI on the debug page to select a file already? Should there be a UI for this in the settings somewhere to make it more "legit"? We'll need a way for the user to tell us about what kind of file they are uploading and that will need some UX'ing thought.

Of the platforms mentioned by John here are some of the exported formats I can find:

I think we should aim for the first 2 (WordPress and Blogger) to be supported first by 0.0.4, and gradually implement others as necessary (or if necessary) while encouraging the community to write the rest based on the example of our first two. We should make our implementations of the WordPress and Blogger import plugins on a base class from the ghost-plugins repo; e.g ImportPlugin to dog food those classes.

So in summary, I think this will put us in a good position for these import plugins:

  1. Get PR #931 merged to fix async doFilter
  2. Decide on a UI/UX for exported file upload that somehow passes along what type of file it is
  3. Incorporate changes from ErisDS/Ghost@5a14d35 but modify import filter to pass file type.
  4. Create new ImportPlugin base class in ghost-plugin repo.
  5. Create WordpressImportPlugin and BloggerImportPlugin inheriting from ImportPlugin. They will check the file type passed to the filter and only do an import for their file type. Other plugins will follow the same pattern.
  6. Profit.

Go Team Go? 🎈

Member

jgable commented Oct 28, 2013

Seems like the import plugin stuff is best discussed in this thread. I took a lot at your importer plugin and I think it will work. We'll need to get my other PR in for the asynchronous doFilter or it won't work when the import actually has to save things to a database.

Is there a UI on the debug page to select a file already? Should there be a UI for this in the settings somewhere to make it more "legit"? We'll need a way for the user to tell us about what kind of file they are uploading and that will need some UX'ing thought.

Of the platforms mentioned by John here are some of the exported formats I can find:

I think we should aim for the first 2 (WordPress and Blogger) to be supported first by 0.0.4, and gradually implement others as necessary (or if necessary) while encouraging the community to write the rest based on the example of our first two. We should make our implementations of the WordPress and Blogger import plugins on a base class from the ghost-plugins repo; e.g ImportPlugin to dog food those classes.

So in summary, I think this will put us in a good position for these import plugins:

  1. Get PR #931 merged to fix async doFilter
  2. Decide on a UI/UX for exported file upload that somehow passes along what type of file it is
  3. Incorporate changes from ErisDS/Ghost@5a14d35 but modify import filter to pass file type.
  4. Create new ImportPlugin base class in ghost-plugin repo.
  5. Create WordpressImportPlugin and BloggerImportPlugin inheriting from ImportPlugin. They will check the file type passed to the filter and only do an import for their file type. Other plugins will follow the same pattern.
  6. Profit.

Go Team Go? 🎈

@jgable

This comment has been minimized.

Show comment
Hide comment
@jgable

jgable Oct 28, 2013

Member

One thought about how we can populate the list of supported import file types is that we can add another filter like supportedImportTypes and each import plugin can add information about their supported file types and a display name.

WordPressImportPlugin = {
  activate: function (ghost) {
    ghost.registerFilter('supportedImportTypes', function (supported) {
      return supported.push({
        name: 'WordPress WXR',
        extension: '.wxr'
      });
    });
  }
}

Then when we render the import page all the available names could be rendered on the page somehow. I'll leave that as an exercise for the reader or something.

Member

jgable commented Oct 28, 2013

One thought about how we can populate the list of supported import file types is that we can add another filter like supportedImportTypes and each import plugin can add information about their supported file types and a display name.

WordPressImportPlugin = {
  activate: function (ghost) {
    ghost.registerFilter('supportedImportTypes', function (supported) {
      return supported.push({
        name: 'WordPress WXR',
        extension: '.wxr'
      });
    });
  }
}

Then when we render the import page all the available names could be rendered on the page somehow. I'll leave that as an exercise for the reader or something.

@mgan59

This comment has been minimized.

Show comment
Hide comment
@mgan59

mgan59 Oct 28, 2013

is there an example of an acceptable import format? I have a django blogging app that I want to export my content out of to import into ghost. So need to know what my json format should look like to import into ghost.

mgan59 commented Oct 28, 2013

is there an example of an acceptable import format? I have a django blogging app that I want to export my content out of to import into ghost. So need to know what my json format should look like to import into ghost.

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 28, 2013

Member

The same format that is output from Ghost's export tool which lives at /ghost/debug/

Member

ErisDS commented Oct 28, 2013

The same format that is output from Ghost's export tool which lives at /ghost/debug/

@mgan59

This comment has been minimized.

Show comment
Hide comment
@mgan59

mgan59 Oct 29, 2013

aahh that helped thnkx :)

mgan59 commented Oct 29, 2013

aahh that helped thnkx :)

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 29, 2013

Member

@JohnONolan Needs to look at the UI for this, but yes it will be somewhere more legit.

One major difference between what we want to achieve and your suggestion, is that the Import Plugin should essentially be one single plugin which supports all of those formats.

Member

ErisDS commented Oct 29, 2013

@JohnONolan Needs to look at the UI for this, but yes it will be somewhere more legit.

One major difference between what we want to achieve and your suggestion, is that the Import Plugin should essentially be one single plugin which supports all of those formats.

@jgable

This comment has been minimized.

Show comment
Hide comment
@jgable

jgable Oct 29, 2013

Member

Ok, I'm working on putting something together for this to work with Wordpress and Blogger first, should have it by tonight my time. I'll put it together as one plugin.

Member

jgable commented Oct 29, 2013

Ok, I'm working on putting something together for this to work with Wordpress and Blogger first, should have it by tonight my time. I'll put it together as one plugin.

jgable added a commit to jgable/ghost-plugin that referenced this issue Oct 29, 2013

ImportPlugin base
- Added an ImportPlugin base after discussion on TryGhost/Ghost#950

@jgable jgable referenced this issue Oct 29, 2013

Merged

ImportPlugin base #1

@jgable

This comment has been minimized.

Show comment
Hide comment
@jgable

jgable Oct 31, 2013

Member

I've started on the Wordpress import piece of this as an example in the ghost-plugin repo. I'm not finished, but I want to give @ErisDS or someone else a chance to take it over if it's not moving along fast enough. There is a spec I've been using to kind of test driven develop it along.

Otherwise, I have Halloween festivities tomorrow so I can't guarantee any progress until the weekend.

Member

jgable commented Oct 31, 2013

I've started on the Wordpress import piece of this as an example in the ghost-plugin repo. I'm not finished, but I want to give @ErisDS or someone else a chance to take it over if it's not moving along fast enough. There is a spec I've been using to kind of test driven develop it along.

Otherwise, I have Halloween festivities tomorrow so I can't guarantee any progress until the weekend.

@mgan59

This comment has been minimized.

Show comment
Hide comment
@mgan59

mgan59 Oct 31, 2013

Do we want to always rely on the tumblr-wordpress conversion from - http://tumblr2wordpress.benapps.net/%E2%80%8E I don't think the original author plans to support this and should anything change this would no longer be a viable means for a tumblr blog to migrate to Ghost.

I have a strange position that my personal site is a django app that syncs/polls tumblr for new entries. I had originally thought I would export my writings from the django app itself to a ghost format since the django database has additional content. But now I'm thinking of just exporting from tumblr directly and will hand merge my other content.

I'd be willing to work on a direct tumblr exporter. I did a quick check there is a nodejs tumblr api for (v2) which requires oauth. I've worked with the v1 tumblr api and could more easily get that working, I'm assuming the tumblr2wordpress is using v1 as well since their is no authentication required.

Thoughts?

mgan59 commented Oct 31, 2013

Do we want to always rely on the tumblr-wordpress conversion from - http://tumblr2wordpress.benapps.net/%E2%80%8E I don't think the original author plans to support this and should anything change this would no longer be a viable means for a tumblr blog to migrate to Ghost.

I have a strange position that my personal site is a django app that syncs/polls tumblr for new entries. I had originally thought I would export my writings from the django app itself to a ghost format since the django database has additional content. But now I'm thinking of just exporting from tumblr directly and will hand merge my other content.

I'd be willing to work on a direct tumblr exporter. I did a quick check there is a nodejs tumblr api for (v2) which requires oauth. I've worked with the v1 tumblr api and could more easily get that working, I'm assuming the tumblr2wordpress is using v1 as well since their is no authentication required.

Thoughts?

@jgable

This comment has been minimized.

Show comment
Hide comment
@jgable

jgable Oct 31, 2013

Member

A straight tumblr exporter would be nice. I haven't started on the tumblr import stuff yet, if you want to take a stab based on the example WordpressImport plugin that would be great. Just beware, not everything has been locked down and could change before we finalize.

Would you plan on being able to get the data out of the tumblr blog from node directly (via a request to their api on the server for instance) or would you have them export it to a file, then upload it through the ghost admin? Right now we're planning on them uploading a file, but we could probably accommodate a straight server side solution with some tweaking.

Member

jgable commented Oct 31, 2013

A straight tumblr exporter would be nice. I haven't started on the tumblr import stuff yet, if you want to take a stab based on the example WordpressImport plugin that would be great. Just beware, not everything has been locked down and could change before we finalize.

Would you plan on being able to get the data out of the tumblr blog from node directly (via a request to their api on the server for instance) or would you have them export it to a file, then upload it through the ghost admin? Right now we're planning on them uploading a file, but we could probably accommodate a straight server side solution with some tweaking.

@mgan59

This comment has been minimized.

Show comment
Hide comment
@mgan59

mgan59 Oct 31, 2013

I only looked a little bit, but I don't think tumblr has an export to file option. I could be wrong. But I think the api is the only export option.

mgan59 commented Oct 31, 2013

I only looked a little bit, but I don't think tumblr has an export to file option. I could be wrong. But I think the api is the only export option.

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Oct 31, 2013

Member

My plan is to rely on the Tumblr > WordPress thing until we have better support for 3rd party integrations. Once we have tools for connecting to other APIs we'll be able to connect to the Tumblr API directly.

Being able to support Tumblr because we support WXR is just a nice little short-term bonus

Member

ErisDS commented Oct 31, 2013

My plan is to rely on the Tumblr > WordPress thing until we have better support for 3rd party integrations. Once we have tools for connecting to other APIs we'll be able to connect to the Tumblr API directly.

Being able to support Tumblr because we support WXR is just a nice little short-term bonus

@jgillich

This comment has been minimized.

Show comment
Hide comment
@jgillich

jgillich Oct 31, 2013

Member

I did not try it, but sky-tumblr-export might be a good starting point.

Member

jgillich commented Oct 31, 2013

I did not try it, but sky-tumblr-export might be a good starting point.

@jwillmer

This comment has been minimized.

Show comment
Hide comment
@jwillmer

jwillmer Nov 4, 2013

Are there any considerations in keeping the former post url for people who want to switch their platform without loosing the linked content?

Associated issue: #1395

jwillmer commented Nov 4, 2013

Are there any considerations in keeping the former post url for people who want to switch their platform without loosing the linked content?

Associated issue: #1395

@ErisDS

This comment has been minimized.

Show comment
Hide comment
@ErisDS

ErisDS Nov 13, 2013

Member

@jwillmer, that is the very reason for the existence of #1395 :)

The repo for this app is here: https://github.com/TryGhost/Ghost-Importer

Going to close this and open relevant issues on that repo.

Member

ErisDS commented Nov 13, 2013

@jwillmer, that is the very reason for the existence of #1395 :)

The repo for this app is here: https://github.com/TryGhost/Ghost-Importer

Going to close this and open relevant issues on that repo.

@ErisDS ErisDS closed this Nov 13, 2013

@arcanis

This comment has been minimized.

Show comment
Hide comment
@arcanis

arcanis Dec 14, 2013

@ErisDS Is it normal the the repo is currently empty ? I have read that there should be some alpha-way to import JSON data, but I can't find them. Have they been removed ?

arcanis commented Dec 14, 2013

@ErisDS Is it normal the the repo is currently empty ? I have read that there should be some alpha-way to import JSON data, but I can't find them. Have they been removed ?

@jgillich

This comment has been minimized.

Show comment
Hide comment
@jgillich

jgillich Dec 14, 2013

Member

@arcanis It is under /ghost/debug/.

Member

jgillich commented Dec 14, 2013

@arcanis It is under /ghost/debug/.

@arcanis

This comment has been minimized.

Show comment
Hide comment
@arcanis

arcanis Dec 14, 2013

Great, thanks ! :)

arcanis commented Dec 14, 2013

Great, thanks ! :)

@bebraw

This comment has been minimized.

Show comment
Hide comment
@bebraw

bebraw Jan 29, 2014

FYI, I wrote a tiny Blogger importer. See blogger2ghost.

bebraw commented Jan 29, 2014

FYI, I wrote a tiny Blogger importer. See blogger2ghost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment