Generate documentation using Docs-Builder

[pomek]

At this moment, works on Docs-Builder are still in progress.

That package is being tested with CKEditor5 all the time, so I've decided to create a ticket for that. It allows creating a branch and pushing the latest changes.

How does Docs-Builder work?

The whole mechanism isn't too complicated but can be a little large. We assume that each repository contains two directories:

1. docs/samples - Sample which are files with extensions: *.html, *.md, *.js. Sample files are similar to manual tests. The *.js files contain part of test and part of code which initializes the editor,
2. docs/guides - Guides which are files with extension *.md.

Before starting docs-builder, we have to collect all these files to one place. Module for collecting files will be used few times:

• From build an editor point of view: collect samples and run them as automated tests (in a local environment),
• From bundler point of view: collect samples, make a configuration object for bundler (based on collected files) and build a bundled editor. It allows testing samples in a production environment.

When all files are collected, docs-builder will do a magic stuff. The whole concept of magic stuff will be described in the future in docs-builder repository.

Related tickets:

• Documentation ckeditor5-design#150
• https://github.com/cksource/docs-builder/issues/1

[pomek]

After extracted packages (from the main repository to smaller ones), I have some troubles with Bundler.

Before my changes, the Bundler worked on hardcoded paths. It didn't allow running the mechanism a few times at the same time. I decided to rebuild it. After some changes, the mechanism could be running multiple times and the built editors won't overrides themselves.

Unfortunately, after extracting to smaller packages, I have to figure out one more time the way for the idea above.

[oskarwrobel]

According to

ckeditor5/dev/tasks/bundle/tasks.js

Lines 43 to 331 in de3325a

	with( configFilePath ) {
	return {
	minify: {
	/**
	* JS minification by UglifyJS.
	*
	* @returns {Stream}
	*/
	js() {
	const bundleConfig = loadBundleConfig( configFilePath );
	const outputFullPath = path.join( bundleDir, `${ bundleConfig.fileName }.js` );
	let stream = gulp.src( outputFullPath )
	.pipe( gulpUglify() );
	return utils.saveFileFromStreamAsMinified( stream, path.dirname( outputFullPath ) );
	},
	/**
	* CSS minification by cssnano.
	*
	* @returns {Stream}
	*/
	css() {
	const bundleConfig = loadBundleConfig( configFilePath );
	const outputFullPath = path.join( bundleDir, `${ bundleConfig.fileName }.css` );
	let stream = gulp.src( outputFullPath )
	.pipe( gulpCssnano() );
	return utils.saveFileFromStreamAsMinified( stream, path.dirname( outputFullPath ) );
	}
	},
	/**
	* Removes all files from bundle directory.
	*
	* @returns {Promise}
	*/
	clean() {
	const bundleConfig = loadBundleConfig( configFilePath );
	const outputFullPath = path.join( bundleDir, bundleConfig.fileName );
	return tools.clean( path.dirname( outputFullPath ), path.join( '' ) );
	},
	/**
	* Combines whole editor files into two files `ckeditor.js` and `ckeditor.css`.
	*
	* For JS bundle is required to pass configuration file `gulp bundle --config path/to/file.js` where
	* we need to specify which editor and features we want to include in our bundle.
	*
	* // example-config-file.js:
	*
	* module.exports = {
	* // Name of CKEditor instance exposed as global variable by a bundle.
	* moduleName: 'MyCKEditor',
	*
	* // Path to specified editor module.
	* // It could be a path relative to `build/esnext/ckeditor5` directory e.g. `editor-classic/classic`
	* // or path relative to directory where build task will be executed `./full/path/to/custom/editor`.
	* //
	* // Note: file extension is appended automatically.
	* editor: 'editor-classic/classic',
	*
	* // List of features.
	* //
	* // Note: file extension is appended automatically.
	* features: [
	* // It could be a plugin name only if plugin is a default CKEditor plugin.
	* 'delete',
	*
	* // It could be a path relative to `build/esnext/ckeditor5` directory.
	* `typing/typing`,
	*
	* // Or it could be a path relative to directory where build task will be executed.
	* './path/to/some/custom/feature',
	* ],
	*
	* // The format of the generated bundle.
	* //
	* // Note: More information about [available formats](https://github.com/rollup/rollup/wiki/JavaScript-API#format)
	* // you can find in Rollup.js documentation.
	* format: 'iife',
	*
	* // Specify path where bundled editor will be saved.
	* //
	* // Note: This path should be relative to `config.BUNDLE_DIR`.
	* fileName: 'ckeditor',
	* };
	*
	* @returns {Promise} Promise that resolve bundling for CSS and JS.
	*/
	generate() {
	const bundleConfig = loadBundleConfig( configFilePath );
	// Create a temporary entry file with proper directory structure if not exist.
	mkdirp.sync( bundleDir );
	const bundleTmpDir = fs.mkdtempSync( bundleDir + path.sep );
	const temporaryEntryFilePath = path.join( bundleTmpDir, 'entryfile.js' );
	fs.writeFileSync( temporaryEntryFilePath, utils.renderEntryFileContent( bundleTmpDir, bundleConfig ) );
	// Bundling JS by Rollup.
	function bundleJS() {
	const outputFile = path.join( bundleDir, `${ bundleConfig.fileName }.js` );
	return rollup( {
	entry: temporaryEntryFilePath,
	plugins: [
	rollupBabel( {
	presets: [ 'es2015-rollup' ]
	} )
	]
	} )
	.then( ( bundle ) => {
	return bundle.write( {
	dest: outputFile,
	format: bundleConfig.format,
	moduleName: bundleConfig.moduleName
	} );
	} )
	.then( () => {
	// If everything went well then remove tmp directory.
	tools.clean( bundleTmpDir, path.join( '' ) );
	} )
	.catch( ( err ) => {
	// If something went wrong then log error.
	gutil.log( gutil.colors.red( err.stack ) );
	// And remove tmp directory.
	tools.clean( bundleTmpDir, path.join( '' ) );
	throw new Error( 'Build error.' );
	} );
	}
	// CSS is already bundled by a build task, so we need only to copy it.
	function bundleCss() {
	const cssSource = path.join( sourceBuildDir, 'theme', 'ckeditor.css' );
	const outputFullPath = path.join( bundleDir, `${ bundleConfig.fileName }.css` );
	const outputDirectory = path.dirname( outputFullPath );
	const outputFileName = path.basename( outputFullPath, '.css' );
	return utils.copyFile( cssSource, outputDirectory, outputFileName );
	}
	// Lets wait for both - JS and CSS.
	return Promise.all( [ bundleJS(), bundleCss() ] );
	}
	};
	},
	clean: {
	/**
	* Removes prepared configs.
	*
	* @returns {Promise}
	*/
	sampleConfigs() {
	return tools.clean( config.DOCUMENTATION.SOURCE_DIR, config.DOCUMENTATION.SAMPLES.CONFIG_DIR );
	},
	/**
	* Removes built editors.
	*
	* @returns {Promise}
	*/
	samples() {
	const editorsPath = path.join( config.DOCUMENTATION.SOURCE_DIR, 'assets', 'scripts' );
	return tools.clean( editorsPath, 'samples' );
	}
	},
	prepareSamplesConfigs() {
	return collectFiles( config, 'SAMPLES', false )
	.pipe( filter( ( file ) => path.extname( file.path ) === '.js' ) )
	.pipe( rename( ( file ) => {
	file.dirname = file.dirname.replace( '/docs/samples', '' );
	} ) )
	// todo: seperate to external method and make unit tests
	.pipe( build.noop( ( file ) => {
	const bundleConfig = {};
	let contents = file.contents.toString( 'utf-8' );
	bundleConfig.format = 'iife';
	bundleConfig.moduleName = contents.match( /([a-z]+)\.create(.*)\/\/ ?editor:name/i )[ 1 ];
	bundleConfig.editor = contents.match( /([-a-z0-9]+\/[-a-z0-9]+)(\.js)?'; ?\/\/ ?editor:module/i )[ 1 ];
	bundleConfig.features = contents.match( /(\[[^\]]+\]),? ?\/\/ ?(.*)editor:features/i )[ 1 ]
	.match( /([a-z-\/]+)/gi );
	bundleConfig.fileName = file.path.match( /ckeditor5-(.*)\.js$/ )[ 1 ];
	file.contents = Buffer.from( `module.exports = ${ JSON.stringify( bundleConfig )};\n` );
	} ) )
	.pipe( rename( ( file ) => {
	file.dirname = file.dirname.replace( /^ckeditor5-/, '' );
	} ) )
	.pipe( gulp.dest( configsDirectory ) );
	},
	generateForSamples( done ) {
	const configFilesGlob = path.join( configsDirectory, '**', '*.js' );
	const configFilePaths = glob.sync( configFilesGlob );
	let index = 0;
	for ( const configPath of configFilePaths ) {
	gutil.log( `Read a configuration from '${ gutil.colors.cyan( configPath ) }'.` );
	const tasksWithConfig = tasks.with( configPath );
	tasksWithConfig.clean()
	.then( () => tasksWithConfig.generate() )
	.then( () => {
	const bundleConfig = loadBundleConfig( configPath );
	const sampleDirectory = configPath.replace( `${ configsDirectory }/`, '' ).replace( /\.js$/, '' );
	const builtEditorPath = path.join( bundleDir, bundleConfig.fileName );
	const destinationPath = path.join( config.DOCUMENTATION.SOURCE_DIR, 'assets', 'scripts', 'samples', sampleDirectory );
	// Copies a editor sources to proper directory.
	return Promise.all( [
	utils.copyFile( `${ builtEditorPath }.js`, destinationPath ),
	utils.copyFile( `${ builtEditorPath }.css`, destinationPath )
	] );
	} )
	.then( () => tasksWithConfig.clean() )
	.then( () => {
	index += 1;
	if ( index === configFilePaths.length ) {
	done();
	}
	} )
	.catch( ( err ) => {
	gutil.log( gutil.colors.red( err.stack ) );
	} );
	}
	},
	register() {
	const tasksWithConfig = tasks.with( args.config );
	gulp.task( 'bundle:clean', tasksWithConfig.clean );
	gulp.task( 'bundle:generate',
	[
	'bundle:clean',
	'build:js:esnext',
	'build:themes:esnext'
	],
	tasksWithConfig.generate
	);
	gulp.task( 'bundle:minify:js', tasksWithConfig.minify.js );
	gulp.task( 'bundle:minify:css', tasksWithConfig.minify.css );
	gulp.task( 'bundle', ( callback ) => {
	runSequence( 'bundle:generate', [
	'bundle:minify:js',
	'bundle:minify:css'
	], () => {
	const bundleConfig = loadBundleConfig( args.config );
	const files = [
	`${ bundleConfig.fileName }.js`,
	`${ bundleConfig.fileName }.css`,
	`${ bundleConfig.fileName }.min.js`,
	`${ bundleConfig.fileName }.min.css`
	];
	const filesStats = utils.getFilesSizeStats( files, bundleDir );
	// Show bundle summary on console.
	utils.showFilesSummary( 'Bundle summary', filesStats );
	// Finish the task.
	callback();
	} );
	} );
	gulp.task( 'bundle:samples:config:clean', tasks.clean.sampleConfigs );
	gulp.task( 'bundle:samples:config', [ 'bundle:samples:config:clean' ], tasks.prepareSamplesConfigs );
	gulp.task( 'bundle:samples:clean', tasks.clean.samples );
	gulp.task( 'bundle:samples', [
	'bundle:samples:clean',
	'build:js:esnext',
	'build:themes:esnext',
	'bundle:samples:config'
	], tasks.generateForSamples );
	}
	};

. I think we should not mix sample generator with bundling. Bundle tasks after extracting to separate repository are no longer a gulp tasks. I understand it as a set of utils for bundling and minifying. In ckeditor5 we register those utils as a gulp tasks, but sample generator could require it and do whatever needs.

I mean something like that:

const bundleTasks = require( 'path/to/bundle/tasks.js' );
let bundles = [];

function createBundle( config ) {
    return bundleTasks.generate( config ).then( bundleTasks.minify( config ) );
}

for ( let sampleConfig of samples ) {
    bundles.push( createBundle( sampleConfig ) );
} 

return Promise.all( bundles );

But first bundle output needs to be configurable :) and you already did it.

Maybe sample generator should be also extracted to separated repository?

[pomek]

The whole problem with building editors for samples has been resolved.

Smaller tasks from ckeditor5-dev-bundler-rollup repository:

• https://github.com/ckeditor/ckeditor5-dev-bundler-rollup/issues/8
• https://github.com/ckeditor/ckeditor5-dev-bundler-rollup/issues/9

Allowed to transform sample for bundle's configuration and use them for bundler.

[Reinmar]

We need to decide how to split the functionality into multiple packages.

The packages:

• dev-compiler
• dev-bundler-rollup
• dev-docs
• dev-tests

The functionality to cover:

1. Creating editor bundle configurations based on passed sample files.
2. Bundling editors:
   1. for release testing purposes,
   2. for production version of documentation.
3. Preparing HTML of samples (e.g. for documentation, JSFiddle, other websites which will want to use those samples):
   1. in dev mode – using source editor,
   2. in production mode – using bundled editor,
4. Building documentation (combining samples, guides and API).
5. Compiling release tests.

Proposal:

1. -> dev-docs. Bundler shouldn't know about it. Will be used by dev-docs (for 2.2.) and dev-tests (for 2.1.).
2. 1. -> dev-tests
   2. -> dev-docs
3. -> dev-docs
4. -> dev-docs
5. -> dev-tests. Though decision because compiler already compile normal tests as this is simply parts of its job. But running release tests require knowledge of the bundler and the compiler should not have it.

Dependency tree:

• dev-compiler: dev-utils
• dev-bundler-rollup: dev-utils
• dev-docs: dev-compiler, dev-bundler-rollup, dev-utils
• dev-tests: dev-docs, dev-compiler (in the future, once it will contain the whole test env), dev-bundler-rollup, dev-utils

[Reinmar]

So, there are 2 packages to create: dev-docs and dev-tests. The second one right now will not be used because it will contain only the code for testing releases, which we created to prove that this is doable at all. We will start using this package when we'll work on porting the testing env to Karma, which will require some additional code.

[Reinmar]

OK, first version of docs builder integration landed on master. Now, the work on the docs builder will continue.