[WORK IN PROGRESS] - List Improvements

[bdbch]

This PR will implement several list item improvements to make handling with lists easier.

Todo

• Correctly handle backspace when at start of list item
  • And no previous list item on the same level exists (lift item)
  • A previous list item exists (join paragraphs)
• Correctly handle delete key when at end of list item
  • And no following list item on the same level exists (merge next content in if possible?)
  • A following list item exists (join paragraphs)
• Handle backspace when the cursor is behind a bulletList (right now the current paragraph gets turned into a list item)
• Fix undoInputrule keymap handler with new backspace behavior

[netlify]

✅ Deploy Preview for tiptap-embed ready!

Name	Link
🔨 Latest commit	2281a22
🔍 Latest deploy log	https://app.netlify.com/sites/tiptap-embed/deploys/648858a915e32e0008042e62
😎 Deploy Preview	https://deploy-preview-3819--tiptap-embed.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[bdbch]

The current new behavior would be the following:

if the cursor is at the start and there is no direct sibling list item on the previous line

in this case, the current list item will just be lifted up one level

From:
- A
  - A.1
    - A.1.a
  - A.2 (we move cursor before A.2 and press backspace)
- B

To:
- A
  - A.1
    - A.1.a
- A.2 (since we pressed backspace on this level, we lifted the item one level up
- B

if the cursor is at the start of a list item and the previous list item is on the same level plus doesn't have a sublist

in this case, the list items plus their paragraphs are merged

from:
- People
  - John
  - Maria
  - Mark (cursor is before "Mark" and we press backspace)
  - Fred

to:
- People
  - John
  - MariaMark (the content from the "Mark" list item gets merged with the previous list item)
  - Fred

in case the merged list item had child items those are lifted up accordingly:

from:
- Recipes
  - Noodles
  - Fruits (lets say we press backspace with a cursor in front of "Fruits")
    - Apple
    - Banana
  - Soups
    - Tomato Soup
    - Noodle Soup

to:
- Recipes
  - NoodlesFruits (noodles+fruits are joined, sub items on the correct position)
    - Apple
    - Banana
  - Soups
    - Tomato Soup
    - Noodle Soup

[bdbch]

Just a quick update:

This PR is almost ready to be merged. Since we have a strict deadline right now for 2.0.0, we're going to release those changes with the first 2.1.0-rc.1 release candidate version.

Reason for that is, that those changes are quite dangerous to get merged fast without testing as a lot of editors could break if some bugs occur. When the 2.1.0-rc.1 version is out, I'd love to invite everyone to test the new list behavior with me, report bugs and issues etc. so we can get list behavior right.

[bkempe]

There seems to be a bug / unintuitive behavior in the preview https://deploy-preview-3819--tiptap-embed.netlify.app/preview/Overview/Installation:

From:
- A
  - A.1
    - A.1.a
  - A.2 (move cursor before A.2 and press backspace 3 times)
- B

To:
- A
  - A.1
    - A.1.a
- A.2 (now we're stuck here and can't backspace to the previous bullet list anymore)
- B

I think the approach for list-items in ReMirror is quite user-friendly:

From:
- A
  - A.1
    - A.1.a
  - A.2 (move cursor before A.2 and press backspace)
- B

To:
- A
  - A.1
    - A.1.a
    A.2 (backspace again)
- B

To:
- A
  - A.1
    - A.1.a
    - A.2
- B

[bdbch]

Thanks for pointing this one out @bkempe - will fix that behavior and get those list changes into 2.1.0-rc.0 by the end of the week!

[kalda341]

Any updates on when this is going to land?

[Nantris]

Further findings I thought should be considered regarding a (I think) previously unknown list copy/paste bug: #3128 (comment)

[Nantris]

Courtesy of @AdventureBeard I tried the extensionized version of these changes with 2.1.0-rc.4.

Tested with this input

<ul>
  <li>
    <p>Item one</p>
    <ul>
      <li>
        <p>Item two</p>
      </li>
      <li>
        <p>Item three</p>
      </li>
    </ul>
  </li>
</ul>

Findings so far:

Unresolved:

1. Pressing Enter with an expanded selection encompassing multiple listItems wrongly inserts a paragraph within the first listItem in the selection (basically another variant of list limbo)

Resolved:

1. Primary backspace issue (list limbo) is resolved

Introduced problems:

1. New backspace issue where you cannot backspace into an existing list unless the last item in the list is only one item deep (otherwise it toggles list on/off indefinitely)
2. New backspace issue where if you start at the end of the last listItem and Ctrl+Backspace until the list is gone, you cannot undo the changes - First undo brings back the bullet list and second undo errors with Uncaught RangeError: Position 12 outside of fragment (<paragraph, bulletList(listItem(paragraph)), paragraph>)
3. The newly reported copy/paste bug is unaffected. A single listItem can still contain multiple paragraph and ul tags

Image for #3

Changes in behavior

1. One of our custom list commands had a minor change in behavior for top-level listItems only. I expect this to be easily resolved but I've not investigated further yet.

Thoughts

I wonder if a ProseMirror plugin to transformPastedHTML could be used to resolve the pasting bug without wreaking havoc somewhere else? I'd not sure how that would interact with user-added plugins that attempt to do the same.

[Nantris]

Friendly bump.

[Nantris]

Regarding this item on the list:

• Handle backspace when the cursor is behind a bulletList (right now the current paragraph gets turned into a list item)

Is this not a more generic problem? It affects all sorts of lists as well as blockquotes. Is this something that can be fixed in a more generalizable way? Or does the (hypothetical) fix need to be applied to each node type individually? @bdbch

I noticed the same sort of issue affects listItem and blockquote when pressing Delete from the last item in a list also.

[Nantris]

@bdbch I implemented your changes as an Extension to try to fix the problems related to Ctrl/Ctrl+Shift in combination with Backspace/Delete. Really this only adds additional key handlers to implement @bdbch's fixes in cases they were not being handled in in the version provided by @AdventureBeard.

I'd encourage everyone interested in this issue to try this extension and report back their findings. Try the extension yourself!

---

I'm not sure why, but some of the issues I was previously seeing no longer seem to be appearing. Perhaps I'm testing improperly, or perhaps I previously tested improperly?

The only issues I now see are:

1. Deleting nodes with children leaves behind empty items - I believe this should instead lift the child items up so that the parents can be deleted. Additionally the resulting selection is incorrectly at the from position instead of the to position

cant-delete-parent-items.mp4

2. Pasting content copied across listItems can still insert extra levels into the list - This actually seems partially mitigated now but still could use an additional check to prevent inserting any more crud than necessary

interstitial-area-pasting.mp4

3. "Handle backspace when the cursor is behind a bulletList (right now the current paragraph gets turned into a list item)" - as mentioned in the original issue.

[kalda341]

@slapbox The extension works great!

One minor issue I've found is that pressing backspace while the cursor is at the first character of the second line of a paragraph in a list item containing multiple paragraphs deletes the previous list item.

IMO a list item containing multiple paragraphs is an invalid state anyway (with this extension the only way I can get get to it is by Ctrl+Shift+V pasting multiple paragraphs into a list item), but it should probably be considered if this state is to be allowed.

[Nantris]

Great find @kalda341. I added a (poorly named) check for that but haven't tested the new version:

  // avoid unindenting at start of paragraph if paragraph is not the first in the listItem
  if (atStartOfSecondOrLaterParagraph(state)) {
    return commands.joinBackward();
  }

This fix will not help users who use the default content rule on ListItem if the second/third/etc node is not a paragraph. I advise people use the following instead of the default:

ListItem.extend({
        content: 'paragraph (bulletList|orderedList|paragraph)*',
})

[Nantris]

I discovered there are some additional remaining problems:

1. Backspace on Android unindents and lifts with a single press of Backspace but it should take two presses to achieve that. And when that problem occurs, the caret is also off-by-one (@bdbch I could really use some advice on possible causes for this)
2. Delete at the end of a node fails to pull the following listItem up and join it to the focused listItem if the following item has a child
3. Enter inside empty child listItem which is neither first nor last among its siblings leads to an extra paragraph inside of listItem

PS: I believe that the best approach to releasing these changes is to release them as something like tiptap-easy-lists-extension. This would guarantee no existing projects would be broken and make it far easier to release the changes for general use.

[kalda341]

@slapbox Yes, I suggest everyone makes the content change - it seems like a much more sensible default.

Those changes have improved things enough for me that I'm going to trial them in production. Wish me luck!

[Nantris]

@bdbch I noticed you've done some more work on this; thank you very much for it!

I know I've commented quite extensively on many related issues here to the point it would be overwhelming to respond to everything, but I hope you could find time to reply to this comment, which I feel is the summation of dozens of hours of investigation into this subject.

Most especially to point 1 and the idea to release these changes as an extension for guaranteed backwards compatibility for all users.

[bdbch]

Hey @slapbox – I've read all your comments on this PR. Good work already. I'll need some time to check out your open issues. I'm heavily packed with a project rn.

I released the first batch of this PR with 2.1.0-rc.9 and will continue working on branch bdbch/list-improvements-v2.

[Nantris]

Thanks for your reply and your hard work @bdbch!

Unfortunately, as released these changes break our existing work and lock us into being stuck at 2.1.0-rc.8. I really think this would be best released as a separate extension, especially if it's being included in a release candidate. I wouldn't expect going from 2.1.0-rc.8 to 2.1.0-rc.9 to introduce any substantial changes, and especially not potentially breaking ones.

I'm not sure right now what would be involved for me to fix our project to work with 2.1.0-rc.9.

As much as I want these changes integrated, I really think that the way they're released right now should be rolled back. I'd totally support a release-candidate release of them via an extension though, because that would be guaranteed not to break anything. Otherwise I'd suggest they wait for 2.2.0 to give people who are already using the 2.1.0 version lineage the opportunity to make it to the stable release and not be locked into a release candidate indefinitely.

Edit: I also think these changes fail to account for Mod-Backspace and Mod-Delete which will inevitably lead to "list limbo" cases appearing.

Edit2: I disabled the addKeyboardShortcuts for ListItem but it still breaks our project for reasons I'm not clear on. Additionally these changes don't address taskItem use cases (which an extension version would automatically be able to handle)

There are also some other cases the gist handled which the released code does not like backspacing away a listItem which has a child - it currently fails and results in a NodeSelection of the parent of the item you try to delete. I won't attempt to outline them all as I'm continuing my work on 2.1.0-rc.8 since that's my only choice at the moment.

[kalda341]

@bdbch I really appreciate you coming back to this - it will be great to have an official solution to these issues.
However, there are a few issues that I've noticed:

• As @slapbox has said, these changes don't work with Mod-Backspace and Mod-Delete + the backspace away of a listItem with a child.
• Backspacing doesn't work correctly with a listItem with multiple paragraph children, which has also been fixed in the extension.
• Backspace works incorrectly at the start of the paragraph after a list. I don't believe anyone has a fix for this yet.

For this reason, I'm going to stick with @slapbox's extension for now instead of upgrading, though I hope these issues can be resolved officially.

Given this is a potentially breaking change for some users this late in an RC, I too support including the fixes as an extension. Including the fixes in 2.2 would be much more palatable, especially if they are a bit more complete and robust.
It's a pity this didn't make 2.0 as that would have been the perfect time to introduce these breaking changes!

Thanks again @bdbch, I really hope we can get to a great solution that works for everyone.

[Nantris]

I found that the problems I thought were interference from 2.1.0-rc.9 were actually a subtle bug I introduced the same day into our project. It does seem that disabling the default commands/keyboardShortcuts for ListItem allows us to use 2.1.0-rc.9. Hooray!

But I still feel that this change should be reverted. It may fix some issues but it generally makes lists even more unpredictable to work with since the partial fixes introduce new edge cases users won't have seen before. I still really think it should be rolled back deferred until 2.2.0.

[bdbch]

Good point. I already thought about reverting this for now.

In the meantime: I created a PR (see #4140) which should make handling positions (specially nested) easier in the future. As soon as I'm done with this, I'll continue work on the lists. In the end I want lists to not be unpredictable anymore and don't break / lock users into unavoidable traps in the content while only navigating with the keyboard.

(this would also be included in 2.1.0 / 2.2.0)

[bdbch]

Backspace works incorrectly at the start of the paragraph after a list. I don't believe anyone has a fix for this yet.

Just made a fix for this in a new branch.

As @slapbox has said, these changes don't work with Mod-Backspace and Mod-Delete + the backspace away of a listItem with a child.

Also made a fix for this in said branch

Backspacing doesn't work correctly with a listItem with multiple paragraph children, which has also been fixed in the extension.

I tried this with a listItem with 2 paragraphs. On backspace, it seems to work fine - just delete is kind of funky. need to look into this. in general this blocks a stable release right now but I also don't really want to revert anything or move it into an optional package.

List behavior on vanilla Prosemirror is really weird and having this as an optional extension would really not fulfill what I'd expect Tiptap to do (I install Tiptap + List extensions and I expect a well working list editing behavior already).

I think I'd rather think about how we can share all that list logic between the list and taskList extensions.

Unfortunately, as released these changes break our existing work and lock us into being stuck at 2.1.0-rc.8. I really think this would be best released as a separate extension, especially if it's being included in a release candidate. I wouldn't expect going from 2.1.0-rc.8 to 2.1.0-rc.9 to introduce any substantial changes, and especially not potentially breaking ones.

Can you give me more information what exactly is breaking right now or even create a new ticket out of this?

cc @slapbox @kalda341

[bdbch]

Maybe @rfgamaral can also drop his 5 cents in here since he pointed me towards this issue. I'd like lists to work reliable but also work as expected from the enduser.

[rfgamaral]

@bdbch The discussion is a bit long already, and I'm not exactly sure what you are asking me. Can you please rephrase?

[bdbch]

I think the main discussion right now is to if the list changes I made should be a separate extension so it doesn't break existing projects or it should be included in the core / list extension itself.

The bugs described in the thread here are already handled as far as I can see.

[Nantris]

@bdbch I think releasing it integrated into the existing lists extension would be fine since it turns out my issues were due to my own mistake - but I don't see how it would work with taskLists in that case which was another reason for my suggestion of a separate extension, or otherwise it seems like there's no way to avoid code duplication.

I'd not release it as part of the existing lists extension until 2.2.0 since it technically would constitute a breaking change, although I think the partially merged changes are still in 2.1.0-rc.x.

List behavior on vanilla Prosemirror is really weird

Tell me about it! I don't understand how it's considered intended behavior since ProseMirror is otherwise so predictable and polished.

[kalda341]

@bdbch 2.2.0 would be great.

[rfgamaral]

I think fixes like this belong in the core extensions. It doesn't make sense to me that I need to install multiple extensions to get lists behaviour working as they should by default.

What problem does the a separate extension solve? Code duplication for task lists? If that's the only problem, I think we should have code duplication for lists and task lists in interim releases (so that we can get these fixes out to everyone ASAP), and iterate on that, perhaps finding a better solution that doesn't rely on an extra extension?

[bdbch]

Code duplication for task lists?

I solved this by moving those helper functions handling the list positioning into the core where it belongs to (as it is used by multiple extension and could be used by other list extensions in the future)

I think we'll stick to it being in core and bump the version higher up so people know about those breaking changes regarding lists and can react accordingly.

[rfgamaral]

I think we'll stick to it being in core and bump the version higher up so people know about those breaking changes regarding lists and can react accordingly.

What kind of breaking changes are we talking about here, @bdbch? Can you give some examples?