Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Conflunce content to markdown #29

Closed
crandmck opened this issue Jul 12, 2016 · 11 comments
Closed

Migrate Conflunce content to markdown #29

crandmck opened this issue Jul 12, 2016 · 11 comments

Comments

@crandmck
Copy link
Contributor

This involves not only duplicating the content, but also getting all the images over. See Migrating LoopBack Docs to Markdown for use with Jekyll.

This is part of Proposal to make LoopBack documentation open source.

@crandmck
Copy link
Contributor Author

crandmck commented Aug 5, 2016

Confluence-space-export-234853-39588.html.zip

This zip file contains an export of the entire APIC space (which is largely duplicated in the LB space). It has the following:

  • attachments folder: contains all the images used in articles. All the images for an article are in a separate sub-folder with a numeric name, and each image file (mostly .png files) also has a numeric name. The file names appear to be unique within the export.
  • images and styles folders: I don't think we need anything in these folders. May might eventually need something in /styles/site.css but I doubt it.
  • An HTML file for each article in the space (I excluded some that we don't need, but I probably missed a couple, so there may be a handful of superfluous files. Hopefully not a problem).

File names are of the form text_rev-number.html. We need to strip off the part starting with underscore (_) because we don't want the Confluence revision number in the title.

For example, Access-token-REST-API_9634160.html would become Access-token-REST-API.html.

There are about a dozen files with purely numeric file names, because the titles include a special character (usually a colon). We may just need to handle these manually, but we'll want to change these files names to something based on the title. For example, 9634117.html has <title>LoopBack in API Connect : Creating a LoopBack iOS app: part two</title>, so it would be called something like Creating-a-LoopBack-iOS-app:-part two.html.

@marc-ed-raffalli
Copy link
Contributor

marc-ed-raffalli commented Aug 12, 2016

The content is ready, just need to clarify the points below:

  • The link rules makes no mention of the anchors which are linking images in the same page. Should we keep them ?
  • Do you have some guidelines for the tables ?
    Some do not seem to be easily maintainable if we convert them to github flavored MD.
  • Some pages have header over 3 levels,
    Should I convert them to ### or leave the original level ?
  • The script detected invalid links (no text or no href), how do you prefer to handle it ?
  • If we keep the top element (LoopBack in API Connect), the side menu will need to support one more level to keep the current structure

@crandmck
Copy link
Contributor Author

crandmck commented Aug 15, 2016

Thanks @marc-ed-raffalli

The link rules makes no mention of the anchors which are linking images in the same page. Should we keep them ?

I'm not sure what these are... I couldn't find an example in the export. Could you give me an example so I can understand what these are?

Do you have some guidelines for the tables ?
Some do not seem to be easily maintainable if we convert them to github flavored MD.

Tables are a bit problematic in MD. For now, let's just keep them in HTML. I think that should work OK at least for a first cut. Later I can look at making some of the simple ones markdown, and adding some custom styles for tables.

Some pages have header over 3 levels,
Should I convert them to ### or leave the original level ?

For now, go ahead and keep the levels, e.g. use #### and on...

The script detected invalid links (no text or no href), how do you prefer to handle it ?

Again, I don't know what these would be.... We might be able to just discard these, but an example or two would help me to understand better what they are.

If we keep the top element (LoopBack in API Connect), the side menu will need to support one more level to keep the current structure

We don't want to keep that. Instead, we want something like this (e.g. in _data/lb_sidebar.yml):

entries:
- title: sidebar
  product: LoopBack
  version: 2.0
  folders:
...

Unfortunately, the Jekyll theme sidebar seems to support only three levels of topics. That covers about 90% of the LoopBack doc structure, but there are a few that go down to a fourth level. I'll have to rework the topic hierarchy a bit to fit those in--that will be a manual process. Until I do that, you can just flatten the fourth-level topics to be children of the grandparent.

One more thing that I forgot when I originally wrote up the conversion rules: We have to support translation into multiple languages (see #33). This has several implications for the conversion script:

  • A new level of folders under pages or each lang: en, fr, ja, and so on. Within these we then have lb2, lb3, and so on.
  • Links have to include the {{page.lang}}, e.g. [LoopBack core concepts](/doc/{{page.lang}}/lb2/LoopBack-core-concepts.html)
  • Article front-matter needs to have language property, e.g.
    lang: en
    Sorry I overlooked this when I first wrote up the spec.

Initially, we'll only convert the English docs, so you only need to worry about en. However, we should be able to use the conversion script for existing doc translations as well, once we get English sorted.

@crandmck
Copy link
Contributor Author

BTW, I just pushed an update to the repo that reflects the changes required to support localization: d35c8d6

@marc-ed-raffalli
Copy link
Contributor

marc-ed-raffalli commented Aug 15, 2016

No problem, it is updated. I just need to finish the file content review (180+ files).

Edited: issue solved, (long day I presume ;) )

I'll post later a report of places where the different issues were, it will help to understand and fix if applicable.

@crandmck
Copy link
Contributor Author

crandmck commented Aug 23, 2016

The initial conversion was accomplished in #34 (thanks @marc-ed-raffalli !)

We (I) still need to:

  • Review and revise the migrated content to fix any glitches. This may take a while.
  • Fix up the nav sidebar; It's limited to 3 levels, but the Confluence sidebar has some 4th-level entries. Need to sort that out somehow.
  • Ensure that all the content from LB and APIC spaces was migrated. I may have missed some pages.
  • Get list of new/updated articles since Aug 4 (when I created the export zip above) and re-conver them to get latest changes. BEFORE doing this, I will "freeze" the Confluence wiki to ensure we don't have to go back and do this again!
  • Migrate content of translated docs.

@crandmck
Copy link
Contributor Author

Confluence-space-export-171136-47671.html.zip

Here is the export of the pages modified since the initial export on Aug. 4.

However @marc-ed-raffalli I cannot get these to convert using your conversion script. I get the following error:

$ node . ../APIC output3
Could not match docs link "API explorer" https://docs.strongloop.com/display/LB/API+Explorer in file Adding-remote-methods-to-built-in-models_10355032.html
Processed page Adding remote methods to built-in modelsCreating, updating, and deleting dataFields filterWhere filterDefining mixinsUsing current contextEventsPreparing for deploymentUsing strong-error-handlermiddleware.jsonModel definition JSON fileCommand-line reference
/Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/src/global-flow/steps/side-menu.js:15
  var sideMenuEntries = structure.hierarchy.children[0].children.map(recursiveAppEntries)
                                                                ^

TypeError: Cannot read property 'map' of undefined
    at module.exports (/Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/src/global-flow/steps/side-menu.js:15:65)
    at nextTask (/Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:5021:18)
    at /Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:5015:17
    at /Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:339:31
    at /Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:847:20
    at /Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/src/global-flow/steps/file-content-conversion.js:21:7
    at /Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:3679:13
    at /Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:952:25
    at iteratorCallback (/Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:997:17)
    at /Users/rand/StrongLoop/Revised docs/confluence-to-jekyll/node_modules/async/dist/async.js:847:20

@marc-ed-raffalli I tried to debug this, but didn't get very far....appreciate any help!

@marc-ed-raffalli
Copy link
Contributor

Will get an eye to it

@marc-ed-raffalli
Copy link
Contributor

The HTML structure is slightly different.
@crandmck It is just a few tweaks to make but once fixed I can merge and send a PR with the files

@crandmck
Copy link
Contributor Author

Awesome, thank you! I'm not sure why the structure is different, but if you can easily fix it, that's great!!

@crandmck
Copy link
Contributor Author

crandmck commented Sep 8, 2016

I'm going to close this as the basic work is done. At this point, it's testing, fixing, refinement, etc, which we'll handle in other issues.

@crandmck crandmck closed this as completed Sep 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants