-
-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RSS Agent Adding Extra Characters to Google Alerts RSS URLs #889
Comments
Hey @marycanady, welcome! Could you show us an example of the Agent configuration that you're using? |
Thanks so much! Here's the code for RSS Agent. I tried it with and without "clean": "true" and "disable_url_encoding": "false", and got same results. { |
Does this happen with non-Google RSS feeds? |
Hey @marycanady, I'll try setting this up locally and see if I can reproduce. I'm pretty swamped for the next two days, though, so it may be later in the week or early next week. If someone else wants to try and debug / fix, go for it! :) |
Apparently FeedNormalizer is doing something wrong. |
When parsing one of these feeds, FeedNormalizer somehow falls back from the rss parser to the simple-rss parser, and the SimpleRSS generates the wrong URLs. require 'simple-rss'
require 'open-uri'
rss = SimpleRSS.parse open('https://www.google.com/alerts/feeds/03898956544915458721/17285894713843133113')
puts rss.items.first.link
#=> https://www.google.com/url?rct=j&sa=t&url=http://www.dakotafinancialnews.com/intrexon-corp-lifted-to-buy-at-zacks-xon/243174/&ct=ga&cd=CAIyGmUxMDlkOTA3MDE0YzQ2ZjY6Y29tOmVuOlVT&usg=AFQjCNH61DNiDuZW20aTxuFzFfNKrlWpVg So, one problem is RSS::Parser failing to parse the feed and another is SimpleRSS not handling XML attributes properly. |
This is the current status, and I'm going out for a meal... |
Thanks! Any updates? It will likely be a significant problem for huginn as so many people use Google Alerts. |
I will try to dig into it this week. |
The first problem was due to feed-normalizer not expecting an Atom feed from RSS::Parser.parse(): https://github.com/aasmith/feed-normalizer/blob/master/lib/parsers/rss.rb#L37 (RSS::RDF has channel() while RSS::Atom::Feed does not) The second "problem" turned out to be a 'feature" of simple-rss, not really dealing with XML at all: https://github.com/cardmagic/simple-rss#usage |
So, it would be the best to enhance feed-normalizer to support Atom, but in the meantime we'd have to monkey-patch it. |
Given aasmith/feed-normalizer#4, I doubt feed-normalizer is being maintained. @knu, does updating it interest you? If not, I'll take a crack at it. |
We could probably use RSS::Parser directly, calling feed-normalizer's HTML cleaner as necessary. |
That might be simpler, if it gives us more control, as I imagine Huginn's RSS parsing needs will only increase with time. |
Thanks guys, I know you're still working on it, but I'd be happy to contribute somewhere for your time, let me know where! |
It's not ideal, but I guess adding a Liquid filter |
@marycanady, would an |
Hi--thanks! I'm a n00b both on Huginn and Git, has this been created, and if so how do I use it? I'm also using Huginn on Heroku, and from what I remember, they don't have the latest snapshot of Huginn (and I'm not yet savvy enough to add new code) so I don't know if I can use it yet. Will look into it soon and let you know. Have you been able to test? |
@marycanady, it has not been merged into Huginn's master branch yet. When it is, you'll be able to update your local checkout of Huginn, then push the update to Heroku. This would be something like |
Thanks--sorry I have very limited time to work on this right now. Can you please tell me when the change has been merged with the master? Can you also please point me to some instructions on how to use the filter? |
@marycanady, @knu's filter has been merged into master. It's a normal Liquid filter, as is documented in our wiki. I think @knu's intention was for you to make an EventFormattingAgent that receives your events, runs the |
FeedNormalizer is no longer maintained, and its Atom support has flaws in that it throws away what RSS::Parser returns and falls back to using SimpleRSS which is not capable of handling XML entities, resulting in getting ususable URLs such as ones including `&`. A breaking change is removal of the `clean` option which needs to be reimplemented if it is important, while I personally think you should use EventFormattingAgent to tweak feed contents if you need to. This should address #889 and #955, among others.
Thanks, I'm finally having time for this now. I had created my Huginn build from a buildpack and I didn't get the admin access set up (had used windows, now trying the VMWare/Linux option, still working on it). If I can't get that to work, will a new instance created using the buildpack have the updates I need to implement the filter? Edited |
Yes, if you're pulling from the current git repo, it should be up to date. |
Thanks, it's working, the Google Alerts URLs are valid. Here's the code for the Formatting Agent, you can close this item. I would suggest adding that no arguments are needed for the unescape filter in the wiki, as I was unclear on this fact. { |
Glad it's working! Would you mind updating the wiki? |
@marycanady, did my JavaScript Agent work? If so, would you mind updating the wiki with your successes, and maybe blogging about it? :) |
Hi--forgive me, but where would I have gotten the JavaScript Agent? |
I emailed you a week or so ago. Just replied to it again. |
FeedNormalizer is no longer maintained, and its Atom support has flaws in that it throws away what RSS::Parser returns and falls back to using SimpleRSS which is not capable of handling XML entities, resulting in getting ususable URLs such as ones including `&`. A new key `links` is added, which lists all `link` elements. A breaking change is removal of the `clean` option which needs to be reimplemented later if it is important. This should address #889 and #955, among others.
I recently checked out feeds generated by Google Alerts and found out they are Atom documents with I think that would be the job of the underlying feed parser which currently is Feedjira, so I'm looking to making a PR for it. We've already been patching Feejira quite heavily, but this would be relatively easy to factor out and feed back. |
Finally, here's the work in progress! |
And this: feedjira/feedjira#424 |
Thanks! I reread the thread and can't remember how I solved the problem...but the work is appreciated! |
RSSAgent is adding "amp;" characters at the end of ampersands in URLs it is outputting, how do I prevent or remove? This happens whether or not I use the "clean:true" option. I know Google Alert URLs are long and complex, but I don't know any alternative. Thanks in advance!
Original:
https://www.google.com/url?rct=j&sa=t&url=http://www.vocalrepublic.com/superbug-threat-prompts-west-to-revisit-soviet-era-virus-therapy/15605/&ct=ga&cd=CAIyGjhiMjMxNTkwMzBiNmY1OGU6Y29tOmVuOlVT&usg=AFQjCNFRgagE8CRhJY-oQqfNzxfRAclTeg
RSS Agent Output:
https://www.google.com/url?rct=j&sa=t&url=http://www.vocalrepublic.com/superbug-threat-prompts-west-to-revisit-soviet-era-virus-therapy/15605/&ct=ga&cd=CAIyGjhiMjMxNTkwMzBiNmY1OGU6Y29tOmVuOlVT&usg=AFQjCNFRgagE8CRhJY-oQqfNzxfRAclTeg
The text was updated successfully, but these errors were encountered: