Skip to content

[FEA] Parallelize documentation post-processing #448

Description

@bdice

Documentation deployment takes more than one hour to run. It may make sense to parallelize the CPU operations that post-process the documentation files. I would guess that this is the slow step that would benefit from parallelization:

for FILE in $(grep "${JTD_SEARCH_TERM}\|${DOXYGEN_SEARCH_TERM}\|${PYDATA_SEARCH_TERM}" -rl \
--include=\*.html \
--exclude-dir=stable \
--exclude-dir=nightly \
--exclude-dir=latest \
--exclude-dir=legacy \
--exclude-dir=cudf-java \
${FOLDER_TO_CUSTOMIZE} ); do
python ${SCRIPT_SRC_FOLDER}/customize_doc.py $(realpath ${FILE})
echo "" # line break for readability
done

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions