composebox_typeahead: Avoid generating broken links. #30071

kuv2707 · 2024-05-13T09:28:54Z

The #**stream>topic** syntax generates broken links for topics containing two backticks or starting/ending with *, because of architectural flaws in the backend markdown processor. So we avoid generating the syntax for such topics and instead generate the normal link syntax in markdown.

Fixes #19873

CZO discussion

Screenshots and Screen Captures

Self-review checklist

Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

Explains differences from previous plans (e.g., issue description).
Highlights technical choices and bugs encountered.
Calls out remaining decisions and concerns.
Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

Each commit is a coherent idea.
Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

Visual appearance of the changes.
Responsiveness and internationalization.
Strings and tooltips.
End-to-end functionality of buttons, interactions and flows.
Corner cases, error conditions, and easily imagined bugs.

kuv2707

I have made a preliminary implementation and highlighted the reasons behind some technical choices.
Once we decide the correct logic to detect faulty topic names, I'll refactor the functions and put them in a sensible place.

kuv2707 · 2024-05-13T09:35:53Z

web/src/composebox_typeahead.js

@@ -995,8 +997,29 @@ export function content_typeahead_selected(item, query, input_element, event) {
            // Stream + topic mention typeahead; close the stream+topic mention syntax
            // with the topic and the final **.  Note that token.length can be 0
            // if we are completing from `**streamname>`.
+            function will_produce_broken_link(topic) {


Currently, I have considered only two cases which produced buggy links:

When there are two or more backticks

When the topic name begins or ends with asterisks.

There can still be topic names which cause broken links. I had earlier tried to parse the syntax text using the frontend markdown parser and see if the link in the parsed html points to the correct stream and topic. But double-backtick topic names gave different urls through frontend and backend markdown processors (in fact, the frontend parser correctly parsed the double-backtick topic name, while the backend processor generated an incorrect url).

This means that checking if the frontend markdown processor parses the syntax correctly gives no guarantee about the correctness of the link in the final message. So I thought the best option would be to check for the individual reported cases which produce broken urls.

I infer from CZO discussions that we will eventually move to a different markdown processor, and this will no longer be an issue then.

web/src/composebox_typeahead.js

alya · 2024-05-14T22:09:44Z

@timabbott Do you want to take a look and provide some feedback here?

timabbott · 2024-05-17T17:54:53Z

web/src/composebox_typeahead.ts

+                    beginning.slice(0, syntax_start_index) +
+                    url_syntax(get_stream_name(beginning), item.topic) +
+                    " ";
+            }


Can you extract this logic to a function and/or add unit tests for it? It's complicated enough that it's very much worth having a test for.

I have extracted the logic and added some unit tests.

kuv2707 · 2024-05-20T08:34:57Z

@timabbott Addressed the suggestion. Please take a look.

timabbott · 2024-05-20T16:40:28Z

web/tests/composebox_typeahead.test.js

+    );
+    assert.equal(
+        ct.stream_topic_link_syntax("#**stream>t", "*asterisk"),
+        "[#stream>*asterisk](#narrow/stream/-1-unknown/topic/*asterisk) ",


Why do all of these have -1 as the stream ID in the URLs? The first couple don't, which makes me think you've got some sort of weird bug in either one test case or the other.

stream ids are only required when we generate the markdown link syntax (since we need to generate the url ourselves).
That is why the first two didn't need them.

web/tests/composebox_typeahead.test.js

kuv2707 · 2024-05-20T17:57:40Z

web/src/composebox_typeahead.ts

@@ -1180,6 +1185,31 @@ export function content_typeahead_selected(
    return beginning + rest;
 }

+function will_produce_broken_stream_topic_link(topic_name: string): boolean {
+    return /(\*+)|(.*`.*`.*)/.test(topic_name);


The links for topic names with double backticks are now generated correctly, but the double backticks cause the enclosed part of the topic name to render as an inline code block.
I don't know what to do in this case since backslash escaping is disabled.

Yeah, I think there's no great answer for that at present. This is #18202.

As long as the link works correctly, though, that's a much less severe issue than #19873 is.

A possible workaround is to use HTML character references, as suggested here: #18202 (comment)
I'd definitely leave that as a separate later commit, though.

gnprice

Thanks @kuv2707! This is an old issue people have repeatedly run into, and I'm happy to see progress being made on it.

web/src/composebox_typeahead.ts

gnprice · 2024-06-03T19:07:25Z

web/src/composebox_typeahead.ts

@@ -1180,6 +1185,31 @@ export function content_typeahead_selected(
    return beginning + rest;
 }

+function will_produce_broken_stream_topic_link(topic_name: string): boolean {
+    return /(\*+)|(.*`.*`.*)/.test(topic_name);


Yeah, I think there's no great answer for that at present. This is #18202.

As long as the link works correctly, though, that's a much less severe issue than #19873 is.

A possible workaround is to use HTML character references, as suggested here: #18202 (comment)
I'd definitely leave that as a separate later commit, though.

web/src/composebox_typeahead.ts

kuv2707 · 2024-06-04T17:44:59Z

@alya Does this PR have to go through the buddy and mentor review stages? (Since it's already reviewed by maintainers once)

alya · 2024-06-05T18:46:18Z

I think it's OK to skip those at this stage. Has all the feedback above been addressed?

kuv2707 · 2024-06-06T15:13:05Z

@alya I have addresed all the feedback.

Changes:

Simplified the regex.
Included $$ in the regex.
Checking both the stream name and the topic name.
Using HTML character references to escape backticks, dollars etc.

I have updated the PR description with the screenshots.

I've initiated a discussion about some decisions regarding corner cases here

timabbott · 2024-06-07T23:08:20Z

web/src/composebox_typeahead.ts

+    const end = syntax.lastIndexOf(">");
+    return syntax.slice(start + 3, end);
+}
+export function replace_markdown_characters(text: string): string {


You should have a blank line between functions.

Also, usually it's better to add new functions above the one function that uses them.

I also think a better name would be something like html_escape_markdown_syntax_characters.

timabbott · 2024-06-07T23:10:00Z

web/src/composebox_typeahead.ts

+    ) {
+        return url_syntax(stream_name, topic_name);
+    }
+    return `#**${stream_name}>${topic_name}** `;


Why is there a trailing space here? Seems like it should be added by the caller if desired.

I noticed in the module that a trailing space was always put after inserting any sort of syntax, so I included it here itself.
Moved the spaces to the caller.

timabbott · 2024-06-07T23:10:48Z

web/src/composebox_typeahead.ts

+    return invalid_stream_topic_regex.test(word);
+}
+
+function get_stream_name(syntax: string): string {


This function name is too generic; something like get_stream_name_from_topic_link_syntax would be more appropriate, though I've not checked exactly how it is used.

This name matches what it does.
Changed.

The #**stream>topic** syntax generates broken links for topics containing two backticks or ending with *, because of architectural flaws in the backend markdown processor. So we avoid generating the syntax for such topics and instead generate the normal link syntax in markdown. Fixes zulip#19873

kuv2707 · 2024-06-08T08:16:30Z

@timabbott I have addressed the feedback.

zulipbot added size: S area: markdown (mentions) Mentions for users/groups/stream/topic/time. bug rust community request Issues of interest to the Rust community labels May 13, 2024

kuv2707 commented May 13, 2024

View reviewed changes

kuv2707 mentioned this pull request May 14, 2024

Compose Box: Implement formatting of channels and topics on paste #29302

Open

12 tasks

alya added the integration review Added by maintainers when a PR may be ready for integration. label May 14, 2024

kuv2707 force-pushed the 19873_topic_name_fix branch from 07cab3f to dbcdfa0 Compare May 17, 2024 10:16

timabbott reviewed May 17, 2024

View reviewed changes

kuv2707 force-pushed the 19873_topic_name_fix branch from dbcdfa0 to 0a026be Compare May 18, 2024 16:11

zulipbot added size: L and removed size: S labels May 18, 2024

kuv2707 force-pushed the 19873_topic_name_fix branch from 0a026be to ff0ba14 Compare May 20, 2024 08:07

kuv2707 requested a review from timabbott May 20, 2024 08:33

kuv2707 force-pushed the 19873_topic_name_fix branch from ff0ba14 to 7455824 Compare May 20, 2024 08:55

timabbott reviewed May 20, 2024

View reviewed changes

web/tests/composebox_typeahead.test.js Outdated Show resolved Hide resolved

kuv2707 commented May 20, 2024

View reviewed changes

kuv2707 force-pushed the 19873_topic_name_fix branch 2 times, most recently from a3c5613 to 8c243b1 Compare May 21, 2024 12:16

kuv2707 requested a review from timabbott May 21, 2024 15:38

gnprice reviewed Jun 3, 2024

View reviewed changes

kuv2707 force-pushed the 19873_topic_name_fix branch 2 times, most recently from 7704b04 to d15f3fd Compare June 4, 2024 12:32

kuv2707 force-pushed the 19873_topic_name_fix branch from d15f3fd to cbc52bb Compare June 6, 2024 15:11

zulipbot removed the size: L label Jun 6, 2024

zulipbot added the size: XL label Jun 6, 2024

timabbott reviewed Jun 7, 2024

View reviewed changes

kuv2707 force-pushed the 19873_topic_name_fix branch from cbc52bb to a436ce8 Compare June 8, 2024 08:15

kuv2707 requested a review from timabbott June 8, 2024 08:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

composebox_typeahead: Avoid generating broken links. #30071

composebox_typeahead: Avoid generating broken links. #30071

kuv2707 commented May 13, 2024 •

edited

kuv2707 left a comment

kuv2707 May 13, 2024 •

edited

alya commented May 14, 2024

timabbott May 17, 2024

kuv2707 May 18, 2024 •

edited

kuv2707 commented May 20, 2024

timabbott May 20, 2024

kuv2707 May 20, 2024

kuv2707 May 20, 2024

gnprice Jun 3, 2024

gnprice left a comment

gnprice Jun 3, 2024

kuv2707 commented Jun 4, 2024

alya commented Jun 5, 2024

kuv2707 commented Jun 6, 2024 •

edited

timabbott Jun 7, 2024 •

edited

kuv2707 Jun 8, 2024

timabbott Jun 7, 2024

kuv2707 Jun 8, 2024 •

edited

timabbott Jun 7, 2024

kuv2707 Jun 8, 2024

kuv2707 commented Jun 8, 2024

composebox_typeahead: Avoid generating broken links. #30071

Are you sure you want to change the base?

composebox_typeahead: Avoid generating broken links. #30071

Conversation

kuv2707 commented May 13, 2024 • edited

kuv2707 left a comment

Choose a reason for hiding this comment

kuv2707 May 13, 2024 • edited

Choose a reason for hiding this comment

alya commented May 14, 2024

Choose a reason for hiding this comment

kuv2707 May 18, 2024 • edited

Choose a reason for hiding this comment

kuv2707 commented May 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnprice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuv2707 commented Jun 4, 2024

alya commented Jun 5, 2024

kuv2707 commented Jun 6, 2024 • edited

timabbott Jun 7, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuv2707 Jun 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuv2707 commented Jun 8, 2024

kuv2707 commented May 13, 2024 •

edited

kuv2707 May 13, 2024 •

edited

kuv2707 May 18, 2024 •

edited

kuv2707 commented Jun 6, 2024 •

edited

timabbott Jun 7, 2024 •

edited

kuv2707 Jun 8, 2024 •

edited