Enhance Notion Data Fetching and Translation Workflow

[luandro]

Description:

We need to update the current Notion data fetching and translation process to streamline the creation and management of translation pages. The updated workflow should ensure that translations are created only when necessary and are properly reviewed before publishing.

Objectives:

• Automatic Translation Page Creation:

  • Create translation pages in Notion automatically for each published English page.
  • Do not create a new translation page if one already exists.
  • Create a new translation page if there is an update to the original English page.

• Workflow Execution:

  • Run the workflow on GitHub Actions when triggered.
  • Allow for user reviewing and editing before publishing the changes.

New Workflow Requirements:

1. Fetching Notion Pages:

   • Fetch published, English Notion pages.
   • Fix the logic for handling section toggles and titles to ensure content is accurately captured.

2. Translation Page Creation:

   • For each English page, check if a corresponding translation page (e.g., for Portuguese) exists.
   • If it does not exist or if the English page has been updated, create a new unpublished translation page in the corresponding language.

3. Translation Process:

   • Translate the content of the page using openai-node (replacing the current usage of ai-markdown-translator).

   • Reference Translator Prompt: Use the following prompt for translation tasks, updated to account for front-matter in Markdown files:

     # Role: Translation Assistant
     
     You are a translation assistant, responsible for translating the text provided by users into the specified language. Please maintain the original format, including any front-matter blocks.
     
     ## Profile
     
     - Language: Chinese  
     - Translation Language: English  
     - Translation Format: Maintain the original format and structure
     
     ## Goal
     
     - Translate the text provided by the user into the specified language while preserving the original formatting.
     - Ensure that front-matter (e.g., YAML metadata enclosed in ---) is correctly recognized. Do not translate metadata keys; only translate the values if appropriate.
     - Avoid translation errors and ensure accuracy in the translated text.
     - Preserve the original format, including code blocks, lists, tables, etc.
     - If the original text includes code, translate only the text and comments within the code into the specified language.
     
     ## Skills
     
     - Proficient in translating multiple languages, including but not limited to Chinese, English, Japanese, Korean, etc.
     - Familiar with the grammar and usage of various languages.
     - Knowledgeable about translation formatting and conventions in different languages.
     - Skilled in translation techniques and methods for multiple languages.
     
     ## Constraints
     
     - Do not translate variable names, function names, class names, or module names within code.
     - Do not translate tags within code blocks.
     - For front-matter, preserve metadata keys and structure. Translate only the corresponding values if needed.
     
     ## Workflow
     
     - Follow these steps for translation:
       1. Read the text provided by the user.
       2. Identify and separate any front-matter from the main content.
       3. Translate the main content into the specified language.
       4. Reintegrate the original front-matter, preserving its format.
       5. Ensure the final output maintains the overall structure and formatting of the original document.
     
     ## Instructions
     
     As a translation assistant, you are ready to translate the text provided by the user while maintaining all original formatting and front-matter details.
     

4. Conversion to Markdown:

   • Use Notion-to-markdown to convert all content to Markdown.
   • Output the English content to the docs folder.
   • Output the translated content to i18n/{lang}/docusaurus-plugin-content-docs.

5. Testing:

   • Begin by modifying the script tests to cover the new workflow.
   • Ensure that all changes are properly tested before deployment.

Steps to Completion:

1. Test Updates:

   • Modify existing script tests to accommodate the new requirements.

2. Script Modifications:

   • Update the scripts to implement the new workflow.
   • Adjust the fetching logic for section toggles and titles.
   • Integrate openai-node for translations using the updated translator prompt.
   • Replace ai-markdown-translator with the new translation mechanism.
   • Incorporate Notion-to-markdown conversion with the proper output paths.

3. GitHub Actions Integration:

   • Configure the workflow to run on GitHub Actions upon triggering.
   • Ensure proper notifications and logs for user review and manual edits before publishing.

4. User Review:

   • Implement a step for users to review and edit the content before final publishing.

Notes:

• The solution should handle cases where translation pages already exist and only update when the English content has changed.
• Ensure robust error handling and logging throughout the process.
• Verify that the GitHub Actions trigger works as expected and that the overall workflow is seamless.

[luandro]

Solved by #7