Skip to content

feat: Mirror all data to a second directory with Obsidian Markdown support#13

Merged
busse merged 8 commits intomainfrom
copilot/mirror-data-to-second-directory
Dec 4, 2025
Merged

feat: Mirror all data to a second directory with Obsidian Markdown support#13
busse merged 8 commits intomainfrom
copilot/mirror-data-to-second-directory

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Dec 4, 2025

  • Add "Sync as Markdown" setting (enabled by default)
  • Convert articles to Markdown with Obsidian frontmatter properties
  • Convert feeds and categories to Markdown
  • Create organized folder structure for Obsidian
  • Add UI toggle in Data Mirror settings section
  • Add proper YAML escaping for security
  • Fix HTML stripping for nested/malformed tags
  • Add tests for new toggle
  • Update README documentation
  • Pass all 19 tests
  • Pass CodeQL security analysis
  • Fix: Preserve line breaks in Summary and Content sections
Original prompt

This section details on the original issue you should resolve

<issue_title>feat: Mirror all data to a second directory</issue_title>
<issue_description>Following the contributor requirements of this repo create a new feature that enables the user to mirror all data from the application into a second identical folder the real intent behind this is to feed this information into a personal knowledge management tool, such as obsidian.</issue_description>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.


Note

Adds a Data Mirror feature that mirrors app data to a chosen directory, optionally converting to Obsidian-compatible Markdown, with full settings UI, IPC handlers, and tests.

  • Settings/UI (renderer):
    • Add Data Mirror section in index.html with directory picker, "Sync as Markdown" toggle (default on), and "Sync Now" button.
    • Render and persist new featureFlags.dataMirror, mirror directory, and syncAsMarkdown; show/hide section based on flag.
    • Wire event handlers for browse/clear/sync; toast feedback.
  • IPC/Main process (main.js):
    • New handlers: select-mirror-directory, get/set-mirror-directory, get/set-sync-as-markdown, sync-to-mirror.
    • Enable default featureFlags.dataMirror: false; trigger background mirror sync after writes to feeds/categories/read-states/settings/articles/AI summaries.
    • Implement syncToMirror():
      • JSON mode: copy *.json files to mirror.
      • Markdown mode: convert feeds/categories/articles to Markdown with YAML frontmatter; organize under feeds/, categories/, articles/{feed-name}/.
    • Utilities: HTML stripping with line-break preservation, YAML escaping, filename sanitization, Markdown builders for articles/feeds/categories.
  • Preload bridge (preload.js):
    • Expose Data Mirror IPC methods to renderer.
  • Docs:
    • Update README.md with Data Mirror feature, Markdown structure, and examples.
  • Tests:
    • Add tests/data-mirror.test.js covering UI toggles, directory selection, sync, and visibility.
    • Update mocks to include dataMirror flag and mirror APIs; adjust keyboard shortcut tests accordingly.

Written by Cursor Bugbot for commit 830ce00. This will update automatically on new commits. Configure here.

Copilot AI and others added 4 commits December 4, 2025 01:29
Co-authored-by: busse <18599+busse@users.noreply.github.com>
- Add feature flag 'dataMirror' to enable/disable the feature
- Add IPC handlers for mirror directory selection and sync
- Add preload.js API methods for mirror feature
- Implement automatic sync on data writes
- Add UI in Settings modal for configuring mirror directory
- Add tests for the data mirror feature
- Update README with documentation

Co-authored-by: busse <18599+busse@users.noreply.github.com>
…rallel file copy

Co-authored-by: busse <18599+busse@users.noreply.github.com>
Co-authored-by: busse <18599+busse@users.noreply.github.com>
Copilot AI changed the title [WIP] Add feature to mirror all data to a second directory feat: Mirror all data to a second directory Dec 4, 2025
Copilot AI requested a review from busse December 4, 2025 01:48
@busse
Copy link
Copy Markdown
Owner

busse commented Dec 4, 2025

@copilot during UAT it was discovered that that data mirroring sync is writing json files, which is correct based on the earlier requirements. Obsidian natively supports Markdown (.md) files, so we need to add a setting option to the feature to "Sync as Markdown (.md)" and have that setting be enabled by default if the parent setting for the sync in enabled. Thusly, when that setting is enabled, the "mirrored" files are converted by the system to Markdown before being synced. The markdown files must have appropriate Obsidian properties https://help.obsidian.md/properties

- Add syncAsMarkdown setting (enabled by default when dataMirror is enabled)
- Convert articles, feeds, and categories to Markdown with Obsidian frontmatter
- Create organized folder structure: articles/{feed}/, feeds/, categories/
- Add YAML frontmatter with title, link, author, pubDate, read status, tags
- Add toggle in Settings UI to switch between Markdown and JSON modes
- Add proper YAML string escaping to prevent injection issues
- Fix HTML stripping to handle nested/malformed tags safely
- Add 2 new tests for the Sync as Markdown toggle
- Update README with detailed Markdown sync documentation

Co-authored-by: busse <18599+busse@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Dec 4, 2025

@copilot during UAT it was discovered that that data mirroring sync is writing json files, which is correct based on the earlier requirements. Obsidian natively supports Markdown (.md) files, so we ne...

Added "Sync as Markdown" option in commit 42b9800.

When enabled (default), data is converted to Markdown files with Obsidian-compatible YAML frontmatter:

  • Articles organized in articles/{feed-name}/ with title, link, author, pubDate, read status, and tags properties
  • Feeds in feeds/ with URL, status, and category info
  • Categories in categories/

The toggle appears in the Data Mirror section of Settings and is enabled by default when the Data Mirror feature is enabled.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • example.com
    • Triggering command: /home/REDACTED/.cache/ms-playwright/chromium_headless_shell-1200/chrome-headless-shell-linux64/chrome-headless-shell /home/REDACTED/.cache/ms-playwright/chromium_headless_shell-1200/chrome-headless-shell-linux64/chrome-headless-shell --disable-field-trial-config --disable-REDACTED-networking --disable-REDACTED-timer-throttling --disable-REDACTEDing-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-REDACTED-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=AcceptCHFrame,AvoidUnnecessaryBeforeUnloadCheckSync,DestroyProfileOnBrowserClose,DialMediaRouteProvider,GlobalMediaControls,HttpsUpgrades,LensOverlay,MediaRouter,PaintHolding,ThirdPartyStoragePartitioning,Translate,AutoDeElevate,RenderD (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title feat: Mirror all data to a second directory feat: Mirror all data to a second directory with Obsidian Markdown support Dec 4, 2025
@busse
Copy link
Copy Markdown
Owner

busse commented Dec 4, 2025

@copilot

image When I view the Markdown files in Obsidian, there are no line breaks in the Summary nor Content

- Convert block-level elements (p, div, h1-h6, li, tr, blockquote) to line breaks
- Convert br tags to single newlines
- Convert hr tags to markdown horizontal rules
- Normalize multiple consecutive newlines to paragraph breaks
- Trim each line while preserving paragraph structure

Co-authored-by: busse <18599+busse@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Dec 4, 2025

@copilot

image

...

Fixed in commit 0131e8b. The stripHtml function now preserves line breaks by converting block-level HTML elements (<p>, <div>, <h1-h6>, <li>, <blockquote>) to paragraph breaks and <br> tags to single newlines before stripping the tags. The Summary and Content sections will now display with proper paragraph formatting in Obsidian.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • example.com
    • Triggering command: /home/REDACTED/.cache/ms-playwright/chromium_headless_shell-1200/chrome-headless-shell-linux64/chrome-headless-shell /home/REDACTED/.cache/ms-playwright/chromium_headless_shell-1200/chrome-headless-shell-linux64/chrome-headless-shell --disable-field-trial-config --disable-REDACTED-networking --disable-REDACTED-timer-throttling --disable-REDACTEDing-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-REDACTED-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=AcceptCHFrame,AvoidUnnecessaryBeforeUnloadCheckSync,DestroyProfileOnBrowserClose,DialMediaRouteProvider,GlobalMediaControls,HttpsUpgrades,LensOverlay,MediaRouter,PaintHolding,ThirdPartyStoragePartitioning,Translate,AutoDeElevate,RenderD (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@busse busse marked this pull request as ready for review December 4, 2025 04:14
@busse busse merged commit 647522b into main Dec 4, 2025
3 checks passed
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the final PR Bugbot will review for you during this billing cycle

Your free Bugbot reviews will reset on January 22

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

Comment thread main.js
const readState = readStates[article.id];
const mdContent = articleToMarkdown(article, feedTitle, readState);
const filename = sanitizeFilename(article.title || article.id) + '.md';
await fs.writeFile(path.join(feedFolder, filename), mdContent, 'utf-8');
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Articles with similar titles can overwrite each other

The sanitizeFilename function can produce identical filenames for different articles, causing data loss. When generating article Markdown files, the filename is derived from article.title || article.id, but if two articles have non-empty titles that sanitize to the same string (e.g., "What: Now" and "What? Now" both become "What-Now.md"), the second article will silently overwrite the first. The article.id fallback only helps when the title is empty/null. Consider appending a unique identifier (like a hash of the article ID or a numeric suffix) to prevent collisions.

Additional Locations (1)

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Mirror all data to a second directory

2 participants