Subscriptions / Publicize: Unsupported shortcodes not stripped from email notifications #869

Closed
csonnek opened this Issue Jul 17, 2014 · 101 comments

Projects

None yet
@csonnek
Member
csonnek commented Jul 17, 2014 edited by jeherve

When a user uses a shortcode that isn't part of Jetpack in a blog post, the actual shortcode is sent via the email notification to subscribers in the body of the email rather than being stripped out. For example:

[dropcap custom_class="normal"]L[/dropcap]orem ipsum...

In the blog post, it looks like this: http://d.pr/i/niwT

But in the email, the full shortcode [dropcap custom_class="normal"]L[/dropcap]orem ipsum... is displayed in the email.

I recommend that shortcodes that aren't supported by Jetpack be stripped in the email notifications to avoid the emails being unreadable for subscribers.

Related differential: D3112-code

Member

The difficulty is that unless we have an index of all registered shortcodes, we can't strip them out, without also stripping out normal content that the user may have in brackets.

Also, even if we sync the list of available shortcodes to wpcom, when using shortcodes with opening and closing tags, should the contents be kept as plain text? Or dropped as 'part of the shortcode'? I'd tend toward the former, and it's certainly better than the status quo.

This is also not explicitly a subscriptions issue, as it also happens for content that displays in the reader, and even via the WordPress.com XMPP server subscriptions.

Member
jeherve commented Jul 18, 2014

That's an old issue, and also affects Publicize. See 4301-wpcom for more reports.

The difficulty is that unless we have an index of all registered shortcodes, we can't strip them out, without also stripping out normal content that the user may have in brackets.

Can't we use something like strip_shortcodes() in this case?

If you'd do that, what would happen with the data of the shortcode which displays a full post content? I'm thinking about the Pods Shortcode i.e. I'm using Pods and the shortcode makes it able for me to set up an automated template system where I've got the data in a database, which can change per post. The shortcode parses everything to and through the template.

Or the Unity webplayer plugin (UnityDog), i.e. which doesn't produce text, but a game made with the Unity Engine being displayed in your browser.

Member
jeherve commented Jul 18, 2014

As far as I know, strip_shortcodes() only removes the shortcode tags, but not the content within the tags.

hmmm... If you say "the content within the tags", what exactly do you mean? Everything between the brackets is being processed? Or you have an "open" tag and a "close tag" like the dropcap shortcode and everything between the open and close tag is shown?

I'm asking it, because I've got plugins installed which don't have an open and close tag like the Dropcap.
What happens if it removed the shortcode tags from something like this (the UnityDog shortcode):
[unity src="622"] which makes the webplayer appear, like here: http://fantasygamecreations.com/2014/07/15/testing-yet-again-a-test-for-the-unity-webplayer/
Or the pods shortcode:
[pods name="gameprojects" slug="76" template="game_full_details"] which produces the content from the pods, like you can see here (everything below the slideshow is from the pods shortcode): http://fantasygamecreations.com/2014/06/28/test-post-for-game-project-information-from-pod/

Member
jeherve commented Jul 18, 2014

[unity src="622"]
[pods name="gameprojects" slug="76" template="game_full_details"]

You won't be able to display content from these 2 shortcodes outside of WordPress, since both shortcodes rely on WordPress and on libraries added by the plugins. So it doesn't make sense to display these shortcodes at all in a notification email or a Publicize excerpt, in my opinion.

You're absolutely right about that. But if I make a post and these codes are in it, they'll automatically be sent to the subscriber if they get the full post instead of a possible manual excerpt, right? Or am I wrong about that?

Member
jeherve commented Jul 18, 2014

But if I make a post and these codes are in it, they'll automatically be sent to the subscriber if they get the full post instead of a possible manual excerpt, right?

You're absolutely correct. That's what @csonnek mentioned above with the dropcap plugin example, and that's what we would need to fix with this issue.

So somehow the fix should be able to distinguish between a shortcode that has actual content and a shortcode that needs WP, the libraries and databases from the plugins..... If Jetpack would give the option to include shortcodes (instead of someone manually having to go through thousands of plugins to check what the shortcode actually does), there would have to be a check or a warning or something where it becomes clear that the shortcode has to have actual content. Or perhaps an option where users can add shortcodes that rely on the content of plugins (like UnityDog and Pods) to exclude thos shortcodes from the notification email or the publicize excerpt.
I don't know if this is convenient and/or userfriendly, but I can imagine myself using it. After all, you;d only have to set it one time at the start of using the plugin, right?

Is there anything plugin authors can do to mitigate this on their side?

Member
jeherve commented Sep 8, 2014

Is there anything plugin authors can do to mitigate this on their side?

I'm afraid not, as this is happening on WordPress.com servers.

Contributor
jkudish commented Sep 24, 2014

The reported issue doesn't affect Publicize or Team Partnerships so I removed those labels. We have it logged elsewhere to deal with shortcodes better within the context of Publicize

The solution I would like to see is run do_shortcode() to the full length post, then strip out all the HTML that are not the allowed HTML email tags. The allowed HTML email tags could be p, a, strong, em, img, and so on.

So if someone has short code that adds disclaimers or other content they would like in the HTML email, it will not get deleted the way that strip_shortcode() would. For my site we have gray boxes around recipes that are made with shortcode, so strip_shortcode() would delete the most important part of the post.

I am curious where the source for the posts are gathered by Jetpack to make the emails. I tried modifying the RSS feed, but that does not seem to be the source.

jenhooks commented Feb 3, 2015

Also reported here: 2076027-t

Member

Also 2094857-t

2108180-t

Contributor

2109247-t

Member

@moraitis - I like this direction

Since the emails get sent from WordPress.com, not your local server, maybe we add a custom field on the post called rendered_content, where we store the content after do_shortcode has been run on it? (And sync that field to wpcom)

Anybody have any thoughts on this idea? Not sure if it's just crazy talk...

2133205-t

2138537-t

Contributor

2169486-t

Hi there, as I have the same problem I would like to know where the post content is requested... One solution would be to provide a filter so that one can define what content shall be used, or in which way the shortcodes shall be handled. Greetings

Member

Also reported here 2186273-t

2207703-t

@jeherve jeherve added the Publicize label Jun 1, 2015
@jeherve jeherve changed the title from Subscriptions: Unsupported shortcodes not stripped from email notifications to Subscriptions / Publicize: Unsupported shortcodes not stripped from email notifications Jun 1, 2015
@kraftbj kraftbj added this to the vFuture milestone Jun 1, 2015
@jeherve jeherve modified the milestone: 3.6, vFuture, 3.7 Jun 10, 2015

Hello, we have this exact problem in with our emails. Unwanted Visual Composer code is showing up un-rendered and making the posts look terrible. Has their been any progress on this issue? I noticed that it's been an issue for over a year as I've looked around for possible solutions.

I got similar issue here. Is it possible to provide filters when Jetpack is preparing the post to be synced so plugin / theme can add support by hooking and modifying the shortcode in post_content?

Or.. maybe parse the content by using the_content filter then kses-ing the output when preparing the post to be synced?

Either way, having this solved will help a lot.

CMIIW anyway: after reading the source code looks like the code responsible for preparing the data to be synced to JP server is Jetpack_Sync->sync(), isn't it?

Contributor
kraftbj commented Jul 3, 2015

Noting that there's an internal team that has this on their radar (how to handle unsupported shortcodes so improve the parity of display between wp.com-served functions and the local JP site). cc: @beaulebens as an FYI of one usage of the ultimate outcome of the team's work.

@kraftbj kraftbj added Mercury and removed Team Partnerships labels Jul 3, 2015
Benunc commented Jul 17, 2015

So, Jetpack pulls straight from the database, or what? Why not pull the post content from the RSS feed, as most shortcodes send processed outputs to the RSS feeds?

I'm still confused where Jetpack gets the content. Straight from the database?

Member
jeherve commented Jul 19, 2015

Straight from the database?

Yes, your post content is synchronized with WordPress.com when you publish the post.

Member
csonnek commented Aug 21, 2015

#2312389-t

@samhotchkiss samhotchkiss modified the milestone: 3.7, Needs Triage Aug 28, 2015
Member

I like the direction that @samhotchkiss has proposed. I'm going to add it to 3.8, so it can be appropriately assigned. I'm not sure why it has the 'Mercury' label. Maybe because it will require more syncing?

@roccotripaldi roccotripaldi modified the milestone: 3.8, Needs Triage Aug 31, 2015
Owner

Yep, we're going to need to handle this via much better syncing of content, specifically in this case, fully-rendered (shortcodes already processed) and then synced. We'll still need to sanitize it on output, and potentially strip some things out on WP.com via kses, but we can get a much better starting point by processing shortcodes and syncing that data over, instead of just the raw content.

Would be good to also perform __() on the content to allow translatable texts....

Member
jeherve commented Aug 31, 2015

@richterd Could you open a new issue for that particular trouble you're having? That seems to be a bit different from the shortcode issue mentioned in this issue.

Thanks!

Member
csonnek commented Sep 13, 2015

Also in #2342763-t

mkrdip commented Oct 15, 2015

I'm adding my two cents, adding manual excerpt to each post resolves this issue temporarily. I tried it with Divi theme from Elegant Themes.

@dereksmart dereksmart modified the milestone: vFuture, 3.8 Nov 2, 2015
@jeherve jeherve modified the milestone: 3.9, vFuture Nov 4, 2015

2454197-t

Member

2481237-t

It seems like a simple solution to all this would be if Jetpack had an intermediate step to review/edit/approve/send this translation of the content before it's cross-posted.

@jeherve jeherve modified the milestone: 4.0, 3.9 Jan 15, 2016
Contributor
jb510 commented Jan 28, 2016

Is there any intent on the JetPack team to fix this so that plugin/theme shortcodes work? or is the only road map getting better at stripping out shortcodes even when they're critical to the content being displayed?

Contributor
kraftbj commented Jan 28, 2016

@jb510 There hasn't been a technical decision made yet. There are some hurdles with returning rendered content (how to sanitize it for e-mail? How do handle column-forming shortcodes in the context of Publicize content?) that we'd need to explore a bit. @beaulebens's comment above stands: #869 (comment)

Owner

Yep, as mentioned above, fixing this with the way that Jetpack currently works depends entirely on improving the way that we sync content to WordPress.com (which is where things like Publicize and email subscriptions actually get sent from).

We're actively working on how to improve that, but don't have a solid plan just yet (there are short and long term considerations to all of the possible approaches we've looked at, so we're trying to figure out something that balances the two). All of that to say yes, we do intend to fix it, and yes, as of right now, there is a team of people actively looking at how to handle this.

Contributor
jb510 commented Jan 29, 2016

TY. I do understand the hurdles and am glad you are intent on working on it (after 18 months wasn't sure there was actually interest).

It seems like it'd be simple to just pull from the RSS feed (which is what I had always assumed occurred before discovering you're pull post_content directly via API). I get it's never that simple, but that's what many similar "post by email" solutions use.

So this has been a problem since 2014 and we're no closer to a solution?!!! I just put up my first post on my new wordpress website using jetpack for my transfered subscribers, and they were sent a huge email full of shortcode! Why was there no warning about this issue before you let me look like an idiot? What is the temporary solution for someone like me, who is not adept at understanding these issue or the solution? Is there another plugin that will perform the same function without these problems? One that recognizes the shortcode or reads from the RSS feed? Or is there a way to send a summary or preview the email and make changes before it sends gibberish to all of my followers? (If they haven't all unfollowed after receiving the unreadable email...)

Member
jeherve commented Feb 2, 2016

So this has been a problem since 2014 and we're no closer to a solution?

We're working on it. You can follow the discussion above to find out more.

Why was there no warning about this issue before you let me look like an idiot?

I'm afraid we do not output any warning about the use of shortcodes in the post editor, sorry about that.

What is the temporary solution for someone like me, who is not adept at understanding these issue or the solution?

Until the problem is fixed, there are 2 alternatives to work around the issue:

  • If you go to Settings > Reading in your dashboard, you can change your Feed Settings to "Summary". Once you save your changes, all subscription emails will only include an excerpt of your post, with a link to continue reading on the site. You can set a custom excerpt thanks to the excerpt box below the editor.
  • If your Feed settings are set to "Full Post", you can split the content of each one of your blog posts in 2 parts thanks to the More tag:
    <!--more-->

Everything that is above the More tag will appear on your home page and in the subscription emails, followed by a link inviting users to click to continue reading; everything that is below that tag will only appear on the post's page. You could consequently place your shortcodes below that tag.

You can read more about it here:
http://en.support.wordpress.com/splitting-content/more-tag/

I hope this helps.

Yes! Thank you so much! This sounds simple enough that even I can handle it.

Thank you for your quick reply!

On Tue, Feb 2, 2016 at 8:33 AM, Jeremy Herve notifications@github.com
wrote:

So this has been a problem since 2014 and we're no closer to a solution?

We're working on it. You can follow the discussion above to find out more.

Why was there no warning about this issue before you let me look like an
idiot?

I'm afraid we do not output any warning about the use of shortcodes in the
post editor, sorry about that.

What is the temporary solution for someone like me, who is not adept at
understanding these issue or the solution?

Until the problem is fixed, there are 2 alternatives to work around the
issue:

  • If you go to Settings > Reading in your dashboard, you can change
    your Feed Settings to "Summary". Once you save your changes, all
    subscription emails will only include an excerpt of your post, with a link
    to continue reading on the site. You can set a custom excerpt thanks to the
    excerpt box below the editor.
  • If your Feed settings are set to "Full Post", you can split the
    content of each one of your blog posts in 2 parts thanks to the More tag:

Everything that is above the More tag will appear on your home page and in
the subscription emails, followed by a link inviting users to click to
continue reading; everything that is below that tag will only appear on the
post's page. You could consequently place your shortcodes below that tag.

You can read more about it here:
http://en.support.wordpress.com/splitting-content/more-tag/

I hope this helps.


Reply to this email directly or view it on GitHub
#869 (comment).

This "Summary" fix did not work at all. The shortcode appears a the very beginning of the email. I must say I'm disgusted. I have a hard time believing the people who developed the JetPack plug-in don't know how to fix this issue. I suspect, you don't want to fix the issue, as that might force people to abandon their self-hosted sites.

Whatever the reason, I see that there is no hope of a solution, since none has been "found" since 2014. However, I will do my best to warn everyone out there who might unwittingly use the JetPack plugin, as I did, without knowing they might drive away their followers with emails full of shortcode instead of content.

Member
jeherve commented Feb 9, 2016

@TamieDearen Both work-arounds appear to work on my end. If that didn't work on your end, could you send us an email with more details about your site, so we can take a closer look?

Thanks!

Whatever the reason, I see that there is no hope of a solution

You can check the discussion above to learn more about the issues behind the bug, and the possible solutions we've discussed. We will find a solution!

Thanks for responding. With the help of some more computer-literate friends in my Facebook group, I have determined a solution that works for now. Essentially, I'm bypassing the editor on my website with Advanced Excerpt (from Delicious Brains) which takes me to the plain wordpress editor for the beginning of the post. This allows me to write a custom excerpt for the emails. After the beginning of the post, I'm actually switching back to my website editor, which gives me a much nicer appearance to the blog post than the wordpress editor. It's complicated and cumbersome, but it's working. Perhaps it will help someone else in my same situation.

Almost two years down the line and this issue is still screwing our those all important emails. The easiest method that has worked for me is to temporarily remove any short-code from first two paragraphs of my post just before hit the publish button. Wait for a moment of two and and revert back to the previous version that has all those short codes.

Thank you. That seems to be working. :D

On Tue, Feb 23, 2016 at 7:32 AM, Pritesh Das notifications@github.com
wrote:

Almost two years down the line and this issue is still screwing our those
all important emails. The easiest method that has worked for me is to
temporarily remove any short-code from first two paragraphs of my post just
before hit the publish button. Wait for a moment of two and and revert back
to the previous version that has all those short codes.


Reply to this email directly or view it on GitHub
#869 (comment).

@TamieDearen - Glad it helped you.

Contributor

2557736-t

2571487-t

2605908-t

2634106-t

Member

2637908-t

Member

2622751-t

2662749-t

Has there been any progress on this bug?

Would you guys consider adding an option to blacklist shortcodes as an interim fix? Or is there a hook or filter I could use (or that could be added) to remove the codes myself before the post content goes up to Automattic?

2674122-t for the better click to tweet plugin

Member
jeherve commented Jun 4, 2016

Also reported in 2680025-t

Contributor
kraftbj commented Jun 5, 2016

For everyone following along, as @beaulebens mentioned earlier ( #869 (comment) ), we need to change quite a bit about how Jetpack passes content to WordPress.com in order to resolve this. We are currently testing the next generation of this process that would give us the building blocks to greatly improve this experience. There isn't anything more firm to share with timelines and whatnot, but did want to let you all know it is still very much being worked on.

@jeherve jeherve modified the milestone: 4.2, 4.1 Jun 9, 2016
Contributor
v18 commented Jun 14, 2016

2699598-t

@jeherve jeherve modified the milestone: 4.3, 4.2 Jul 6, 2016
Member
RCowles commented Jul 6, 2016

Also reported in:
2724287-t

@richardmuscat richardmuscat modified the milestone: 4.3, 4.4 Jul 7, 2016
Member
RCowles commented Jul 8, 2016

2727001-t

Another report: 2747161-t

2748670-t

2821321-t

lamdayap commented Sep 9, 2016

1272800-t

2862547-t

2862411-t

2909535-t

2913905-t

fransimo commented Dec 8, 2016

+1

2952616-t

2948323-t

pmciano commented Dec 20, 2016

2958794-t

Contributor

This should now be 99.9% fixed. We are handling 200+ popular shortcodes in the email sending routines in WPCOM.

Contributor
gravityrail commented Jan 13, 2017 edited

And with the sync shortcode tags above coming out in JP 4.5, that should get us to 99.99%. (100% is not possible since it's impossible to always distinguish between a real shortcode and a word in [brackets])

3138886-t

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment