[Export/Import] Block-Level Back-References Are Not Preserved

[pmbauer]

Describe the bug

Restoring a graph from a backup does not preserve block-level back-references.

To Reproduce

Download this minimal repro graph and extract.
Roam-Export-1595945469902.zip.

It contains two pages: [[Concept A]] [[Concept B]]

[[Concept A]] contains a block reference to a block in [[Concept B]], text Lorem ipsum

Steps to reproduce the behavior:

1. import the minimal repro graph into any roam database
2. Go to [[Concept B]]
3. We expect a button next to the block with the count "1". There is a page-level back-reference to [[Concept A]] but the block level back-reference on the block Lorem ipsum is missing.
4. Go to [[Concept A]] and edit the block containing the block reference to Lorem ipsum (e.g. add a space and delete it).
5. Go to [[Concept B]]. Observe the previously missing block-level back reference is present.

System Information:

• Roam version: [0.7.2-6a07d96e]
• Device: iPhone 11 and macbook pro
• OS: iOS and MacOS
• Browser: Safari and Brave

Additional context
Export/Import related bugs

• Block-Level Back-References Are Not Preserved
• Node Collapse State is Not Preserved
• View As State Is Not Preserved
• Filters Are Not Preserved
• Nodes with Versions are Not Preserved

This also impacts calculations. After importing blocks with calculations, the computed values are wrong.
You have to make a minor edit to every calculation block to make it show the correct value.

[joelchan]

This is a serious issue, thanks for raising, @pmbauer. Collapse states and views and filters I can live with not being preserved, but given that one fundamental differentiator in Roam is block-level referencing, breaking that is a critical gap imo.

[joelchan]

Confirmed that I was able to replicate this. Right after import, block reference is present on [[Concept A]] from [[Concept B]], but missing in [[Concept B]]

After editing the block in [[Concept A]] that contains the reference, the backlink indicator now shows up for the block in [[Concept B]]

[pmbauer]

Thank you for replicating the issue.
I was exporting just a small subset of my graph with hundreds of block-level references interwoven in intricate patterns and losing the back links makes the public database significantly less valuable and more difficult to navigate.

And the loss of collapsed state makes visually surveying notes especially difficult. Losing View As State is an annoyance in comparison.

[pmbauer]

This also impacts calculations. After importing blocks with calculations, the computed values are wrong.
You have to make a minor edit to every calculation block to make it show the correct value.

[spikedom]

This issue also affects drawings. These are not preserved across imports and exports.

[pmbauer]

yep; releasing a new feature without export support is very half-baked https://twitter.com/pmbauer/status/1309227917285756932

[filipesilva]

Heya all

We've recently (today) added a new high fidelity restore option. This lets you restore (not import) a graph from a high fidelity edn backup (a new backup format we added last week). You can find it in ... -> graph settings -> restore.

Now, please bear in mind that this essentially means deleting all of your data and then adding the data from the backup. We recommend that you test restoring to a new graph beforehand to make double sure nothing goes wrong. If you run into trouble with the restore feature, let me know and I'll jump on it.

Along with this feature we also fixed a problem where "Check for updates" wasn't working at all, so you'll have to manually close all tabs and reopen one to get the new feature. Today's version that includes both these things is 0.7.4-06236d67.

[pmbauer]

@filipesilva The motivating use case for this issue was exporting subsets of a graph. Should I open up a new issue since this is now considered "solved"?

[pmbauer]

#588

[filipesilva]

The mechanism by which we're supporting this sort of high fidelity exports is the EDN export format. Right now we only have EDN export when using "Export all", and we only support importing EDN exports using "Restore Graph".

Given this context, I think what you want is EDN exports for single pages and importing EDN exports outside of restore graph. Those two things would be feature requests, since at the moment we're not thinking of providing such high fidelity to either markdown or JSON exports.

[pmbauer]

@filipesilva however you want to classify it, existing json and markdown exports still have these bugs

[pmbauer]

@filipesilva I'm not asking for EDN exports for single pages, although that would be nice. I'd just like the existing All Pages or single page exports and subsequent imports to work and they don't.

[filipesilva]

@pmbauer it is true that both markdown and json exports don't support the full range of Roam functionality. Different formats have different levels of fidelity, and the way we've found to add a high fidelity export was a new format, the EDN one.

It's still true that the new format doesn't support single page export, or general purpose import (instead of restore). I'm not sure when/if this will be something we do. The best I can do here is to confirm that is indeed the case. You might have success restoring to a new graph and copy pasting from one graph to the other (inter-roam copy paste is usually pretty high fidelity), but that'll probably not cover all cases.

You can frame that as "the existing formats still don't do what I expect" instead of "the way to do what I expect is with the new format", but the former isn't really a bug since we don't have the intent to backport this level of fidelity to the markdown and json.

To be clear here, to me a bug is when something doesn't work the way it was designed to work. The intent was there, but it the execution did not follow through correctly. The markdown and json exports were never designed to be high-fidelity, they were designed to preserve enough information that they could be used outside of Roam.

I don't think you're going to agree with me on what whether this is a bug or not. I'm just trying to give you my point of view here to help in setting the expectation that we're not planning on adding the level of fidelity you expect to markdown or json exports.

[joelchan]

@filipesilva This is great news! If I understand correctly, this means I can now do a full backup and restore of a graph, but I still cannot losslessly import (subsets of) other graphs. The former is a welcome fix (will start doing periodic EDN backups), but the latter is still something I want. But I understand if that is more of a feature request to you; I'll open a feature request for that.

[pmbauer]

you can frame that as "the existing formats still don't do what I expect"...
The markdown and json exports were never designed to be high-fidelity

The Roam team has in the past very publicly promoted the import/export feature for multiple use cases including backups and selective import along with claims markdown/json were equivalent. To that extent your framing that they were designed to be low-fidelity export-only formats all along is very surprising.

No matter. I'll adjust to the new reality. I'm happy to recognize my expecting selective import to work was unsound so long as it's recognized it's a much needed feature and can get some priority in the roam backlog, regardless of format specifics.

[pmbauer]

@filipesilva where is the best place to document feature requests so the Roam team will see it? In the past it was the forum, but based on discourse stats it doesn't look like the roam team has eyeballs there.

[twleung]

@filipesilva I assume that all this functionality works if the graph is a local graph?

[filipesilva]

@pmbauer checked with folks and https://github.com/Roam-Research/issues#feature-requests seems to be up to date, the forum should be the place for feature requests.

@twleung it should, yes. If you find something that doesn't on local graphs, let me know.

[cweekly]

@twleung YMMV but I just migrated from hosted -> local graph, via EDN export (of hosted) + restore (in local, from hosted's EDN). It worked perfectly. I followed that up with EDN export and restore (all from local) and again, I'm not seeing any problems.

EDIT: 2021-02-08 I'm experiencing severe performance issues with my local graph; it takes more than a minute to open it, triggering browser dialogs asking if I want to continue to wait. Beyond disappointed. I so want Roam to work well enough in offline | local mode to continue to be the hub of my digital notes and workflow. But it's just not. :(