Releases: snarfed/granary
Releases · snarfed/granary
v6.2:
Notable changes
as1
:get_owner
bug fix forpost
,update
,delete
activities.activity_changed
: add newinReplyTo
kwarg.is_public
: add newunlisted
kwarg.
as2
:to_as1
: bug fix, preserveobjectType: featured
for banner/header images even whenmediaType
is also set.is_public
: add newunlisted
kwarg.from_as1
:- For
icon
field, prefer image types that are allowed by Mastodon. - Bug fix, handle
stop-following
with stringobject
id.
- For
atom
:- Add new
extract_entries
function. activity_to_atom
: default actor/author name to username.atom_to_activities
: support top-levelentry
element as well asfeed
.atom_to_*
:- add
object.author
- default
objectType
toarticle
/note
andverb
topost
- convert
link rel=self
/alternate
tourl
- use
displayName
in objects instead oftitle
- Interpret entry
link
withoutrel
as self link.
- add
- If
entry.author
doesn't have id or url, default them to feed author's.
- Add new
bluesky
:- Implement
create
andpreview
. - Fully support both
record
andobject
types infrom_as1
andto_as1
. Useto_as1
'stype
kwarg andfrom_as1
'sout_type
kwarg to disambiguate. - Implement
Bluesky.post_id
. - Add new
blob_to_url
function. - Delete
as1_to_profile
, switchfrom_as1
to return$type: app.bsky.actor.profile
. - Convert HTML
summary
andcontent
to plain text. - Implement
Bluesky.user_to_actor
,Bluesky.get_actor
. - Don't log in (fetch an access token) eagerly in the constructor; wait until the client makes a call.
- Prefer DID to handle in API calls that accept either.
at_uri_to_web_url
: support lists.web_url_to_at_uri
: convert profile URLs likehttps://bsky.app/profile/snarfed.org
to profile record URIs (at://snarfed.org/app.bsky.actor.profile/self
) instead of repo URIs (at://snarfed.org
).- Add
from_as1_to_strong_ref
. - Allow
:
s in record keys (atproto#2224). to_as1
:- Convert blobs, both new and old style, to PDS
getBlob
URLs. - Add new
uri
kwarg. - Translate
handle
tousername
, add newrepo_handle
kwarg. - Add support for
app.bsky.feed.repost
,app.bsky.graph.defs#listView
,app.bsky.feed.defs#blockedPost
. - Add
actor
/author
based onrepo_did
. - Improve
url
field: include custom handles, only userepo_did/handle
forapp.bsky.actor.profile
. - Handle bad facet indices that point inside Unicode code points (example; discussion).
- Convert
!no-unauthenticated
label on profiles to AS1@unlisted
audience target (bridgy-fed#828).
- Convert blobs, both new and old style, to PDS
from_as1
:- Add
out_type
kwarg to specify desired output type, egapp.bsky.actor.profile
vsapp.bsky.actor.defs#profileViewBasic
vsapp.bsky.actor.defs#profileView
. - Add
blobs
kwarg to provide blob objects to use for image URLs. - Add
client
kwarg to fetch and populate CIDs. - Handle mention tags pointing to bare DIDs.
- Use
parent
asroot
in replies. (Technically wrong in cases where the parent isn't the root, but we don't actually know the root. 🤷) - Bug fix: handle bare string URLs in
image
field. - Bug fix: handle tags without
url
field. - Strip trailing slash from home page URLs in order to remove visible
/
from rel-me verified links on Mastodon etc. - Convert
attributedTo
to singular if it has only one element. - If
name
isn't set, fall back topreferredUsername
or infer Webfinger handle fromid
orurl
. - Prioritize bsky.app profile URL before handle URL in
url
field (bridgy#1640). - Convert
bsky.app
inReplyTo
URLs toat://
URIs. - Tighten up
datetime
conversion to match the ATProto recommended format.
- Add
- Implement
facebook
:- Remove
Facebook.fql_stream_to_post
. Facebook turned down FQL in 2016.
- Remove
github
:- When converting data to AS1, use
displayName
in objects instead oftitle
.
- When converting data to AS1, use
mastodon
:get_activities
bug fix: use query params for/api/v1/notifications
API call, not JSON body.- Convert HTTP 200 responses with
error
JSON field (eg from Sharkey) to 400/401 exceptions. - Prefer
media_attachments.remote_url
when available since it may be more long-lived thanurl
for remote statuses (bridgy#1675).
microformats2
:object_to_json
bug fix: handle singularinReplyTo
.json_to_object
bug fix: handle list-valuedlocation
.
nostr:
get_*
: return partial results when the websocket connection is closed prematurely.to_as1
: handle invalid NIP05 values (eg{}
)
rss
:to_activities
:- Use
objectType: note
iftitle
isn't set or is a prefix (possibly ellipsized) ofcontent
/description
. - Add support for images in
media:content
tags (#674).
- Use
Source
:postprocess_activity/object
: addmentions
kwarg to convert @-mentions in HTML links tomention
tags.
v6.1:
Notable changes
Nostr, Bluesky get_activities
, lots of improvements in as2
and microformats2
, and more!
REST API breaking changes:
Twitter is dead, at least in the REST API.
Non-breaking changes:
- Add new
nostr
module! as1
:- Add
get_owner
,targets
. - Add
accept
,reject
,stop-following
toVERBS_WITH_OBJECT
and removerepost
, it's not an AS1 verb. - Handle
url
field list values (even though it's invalid AS1).
- Add
as2
:to_as1
:- Improve
Video
handling: supportLink
objects inurl
, extract stream URLs and types from linktag
s. - Coerce non-float
latitude
andlongitude
to float, raiseValueError
on failure. - Put image attachments into
image
as well asattachments
(bridgy-fed#429). - Handle Hubzilla's composite object attachment
value
s. - Bug fix for null
mediaType
inattachment
andtags
.
- Improve
- Add new
TYPES_WITH_OBJECT
constant. - Add new
get_urls
,address
functions. - Improve
Content-Type
compatibility withapplication/ld+json; profile="https://www.w3.org/ns/activitystreams"
. - Bug fix for
Undo
activities with bare string idobject
s. - Revise HTML in
PropertyValue
attachments on actors to include full URL in anchro text to be compatible with Mastodon's profile link verification.
atom
:activities_to_atom
etc:- Switch
content
from XHTML to HTML inside CDATA to support non-XHTML input content (bridgy-fed#624. - Bug fix, handle bare string URL
image
values. - Bug fix, emove incorrect
type="application/atom+xml"
fromrel="self"
link
inentry
. - Render
objectType: comment
attachments. - Remove invalid
<a>
element for tags.
- Switch
- Bug fixes in
activity_to_atom
/activities_to_atom
for dict-valuedurl
fields. - Render images in article/note attachments.
- Render
objectType: service
attachments, eg Bluesky custom feeds.
bluesky
:- Implement
Bluesky
API class, includingget_activities
. - Drop bundled
app.bsky
/com.atproto
lexicons, use lexrpc's instead. - Convert reposts, quotes, inline links, attached links, and mentions, both directions. Includes Bluesky facet (rich text) support.
- Handle quote posts with attached images, both directions.
- Handle likes, both directions.
- Add new
web_url_to_at_uri
function. from_as1
: handle link tags without start/end indices.to_as1
:- Add new
type
kwarg. - Generate staging.bsky.app profile and post URLs.
- Propagate profile
did
into actorid
. - Add unimplemented stub for custom feeds, eg
app.bsky.feed.defs#generatorView
.
- Add new
- Add
as1_to_profile
. - Bug fix for converting follows, both directions:
subject
inapp.bsky.graph.follow
is followee, not follower. (That field is badly named!)
- Implement
jsonfeed
:activities_to_jsonfeed
:- Bug fix, handle bare string values for
image
andstream
. - Bug fix: handle non-object
author
.
- Bug fix, handle bare string values for
mastodon
:status_to_object
: add/fix alt text handling for images.
microformats2
:json_to_html
:- HTML-escape tag and quote attachment names. Fixes GHSA-4w4f-g49g-3f7j; thank you @janboddez!
json_to_object
:- Improve handling of items with multiple types by using post type discovery more aggressively.
- Normalize ISO-8601 format of
published
andupdated
timestamps.
object_to_json
:- Bug fix, handle bare string URL
image
values. - Normalize ISO-8601 format of
published
andupdated
timestamps. - Handle bare string ids for
replies
andshares
(usually from AS2.)
- Bug fix, handle bare string URL
- Include
objectType: service
attachments, eg Bluesky custom feeds, in JSON and HTML output.
rss
:from_activities
: handle bare string idauthor
.
v6.0
Breaking changes
as2
:- Interpret bare string
object
,inReplyTo
, etc values as ids, convert them to bare strings orid
instead ofurl
.
- Interpret bare string
microformats2
:- Convert simple string
in-reply-to
,repost-of
,like-of
etc values to AS1 bare strings orid
s instead ofurl
s.
- Convert simple string
Non-breaking changes
- Add new
bluesky
module for Bluesky/AT Protocol! as1
:- Add the
organization
object type andACTOR_TYPES
constant (based on AS2). - Add new
get_ids
,get_object
, andget_objects
functions.
- Add the
activity_changed
: ignoreinReplyTo.author
(snarfed/bridgy#1338)as2
:- Support converting between AS1
stop-following
and AS2Undo
Follow
. - Support AS2
Accept
andReject
for follows as well as event RSVPs. - Add support for the
Question
(ie poll),Organization
, andDelete
object types. - Convert
to
/cc
to/from AS1to
for public and unlisted. - Handle
type: Document
video attachments like Mastodon emits. from_as1
: bug fix for image objects withurl
andvalue
fields (for alt text).from_as1
: bug fix, handle bare string URLimage
values.from_as1
: converturls.displayName
toattachment.name
(bridgy-fed#331).from_as1
: preserveinReplyTo
object values as objects, inline single-element lists down down to just single element.to_as1
: useobjectType: featured
for first image inimage
field.to_as1
: populateactor
intoobject.author
forUpdate
s as well asCreate
s.to_as1
: convert Mastodon profile metadataPropertyValue
attachments tourl
composite objects withdisplayName
.- Preserve
to
andcc
values when converting both directions.
- Support converting between AS1
atom
:- Bug fix for rendering image attachments without
image
field to Atom. - Bug fix for
published
andupdated
in entries with objects, eg likes, reposts, RSVPs, bookmarks. Thanks @gregorlove! (#480) - Bug fix for content
activity/ies_to_atom
whenobject
is present and empty. - Bug fix for objects with elements without
objectType
in theto
field.
- Bug fix for rendering image attachments without
flickr
:get_activities
: add support for thecount
kwarg.
github
:get_activities
: add support for thecount
kwarg.
jsonfeed
:- Switch from
white-space: pre
CSS to converting newlines to<br>
s because some feed readers follow it strictly and don't even line wrap (#456).
- Switch from
mastodon
:- Add compatibility support for Truth Social.
- Handle truncated JSON API responses.
microformats2
:json_to_object
: drop backward compatibility support forlike
andrepost
properties. Background discussion.json_to_object
: add newrel_urls
kwarg to allow attachingdisplayName
s tourls
based on HTML text ortitle
attribute (bridgy-fed#331).- Add new
json_to_activities
function. hcard_to_html
/maybe_linked_name
: whenname
is missing, use pretty URL as visible text.- Support the
h-card
org
property. json_to_object
: handle compositersvp
property value.json_to_object
: bug fix whenfetch_mf2
is True, handle when we run the authorship algorithm and fetch an author URL that has au-photo
withalt
.
rss
:from_activities
: fix item ordering to match input activities.
v5.0:
Breaking changes
- Drop Python 3.6 support. Python 3.7 is now the minimum required version.
- Twitter, Instagram, Mastodon:
- Drop
get_activities
cache
kwarg's support for App Engine memcache interface. It's now only used as a plaindict
.get_activities
will now make many small modifications, so if you pass an object that implements those as API calls, you'll probably want to batch those separately.
- Drop
- Twitter, Mastodon, Flickr, GitHub:
create
/preview
: support the AS1favorite
verb as well aslike
. (bridgy#1345)
- Atom:
- Switch to converting AS1
id
(instead ofurl
) to Atomid
.
- Switch to converting AS1
- Reddit:
- Implement
get_actor
.
- Implement
- Mastodon:
create
/preview
: allow non-Mastodon replies, ie activities that includeinReplyTo
URLs even if none of them point to a toot. (bridgy#1321)- Raise
requests.HTTPError
withresponse.status_code
502 instead ofJSONDecodeError
on non-JSON responses. This is synthetic, but more helpful for error handling.
- microformats2:
object_to_json
and related functions: handle all escaped HTML entities, not just&
<
>
.- Unify
microformats2.prefix_image_urls
andprefix_video_urls
into a newas1.prefix_urls
function.
- RSS:
- Remove
itunes:category
. It has to be one of Apple's explicit categories, which we aren't prepared to validate, so don't try.
- Remove
- ActivityStreams 2:
- Translate both
url
andurls
from AS1 into multi-valued AS2url
field.
- Translate both
- Move a number of utility methods from the
Source
class to a newas1
module:object_type
,merge_by_id
,is_public
,add_rsvps_to_event
,get_rsvps_from_event
,activity_changed
,append_in_reply_to
,actor_name
,original_post_discovery
. as1.original_post_discovery
: remove deprecatedcache
kwarg.
Non-breaking changes
- ActivityStreams 2:
- Fix spec compliance bug:
icon
andimage
are singly valued, not multiply valued. - Add new
is_public
method andPUBLIC_AUDIENCE
constant. - Prefer
"objectType": "featured"
first in theimage
field when converting from AS1, last in theicon
field. This matches the ActivityPub (Mastodon) convention of usingicon
for profile pictures andimage
for header images. - Propagate
url
values into newPropertyValue
attachments onPerson
objects; these end up in Mastodon's "profile metadata" link fields. to_as1
: if an attachment'smediaType
isimage/...
, overrideobjectType
and set it toimage
.
- Fix spec compliance bug:
- Twitter
- Trim alt text in line between post preview and creation
- Correctly trim Twitter alt text
- Facebook
- Scraping: extract post id and owner id from
data-ft
attribute and_ft_
query param more often instead ofstory_fbid
, which is now an opaque token that changes regularly. (facebook-atom#27)
- Scraping: extract post id and owner id from
- Instagram
- Add new
Instagram.scraped_json_to_activities
method.
- Add new
- GitHub
create
andpreview
: convert profile URLs to @-mentions, eghttps://github.com/snarfed
to@snarfed
(bridgy#1090).get_activities
withactivity_id
now supportsfetch_replies
andfetch_likes
.
- Reddit
- Add
cache
support toget_activities
.
- Add
- REST API
- Add new
/scraped
endpoint that acceptsPOST
requests with silo HTML as input. Currently only supports Instagram. Requiressite=instagram
,output=...
(any supported output format), and HTML as either raw request body or MIME multipart encoded file in theinput
parameter.
- Add new
- microformats2
- Add new
extra
andbody_class
kwargs toactivities_to_html
. - When converting
u-featured
images to AS1, add new non-standard"objectType": "featured"
field to distinguish them fromu-photo
. - Convert
p-note
to AS1summary
. - Bug fixes for converting
image
attachments tophoto
.
- Add new
Source.original_post_discovery
: add newmax_redirect_fetches
keyword arg.
v4.0:
Breaking changes
- Drop Python 3.5 support. Python 3.6 is now the minimum required version.
Non-breaking changes
- Add new
include_shares
kwarg toget_activities
, implemented for Twitter and Mastodon. Defaults toTrue
. IfFalse
, shares (retweets in Twitter, boosts in Mastodon) will be discarded and not returned. Also add a correspondingshares
query param to the REST API. - RSS:
- Add support for RSS input via new
rss.to_activities
function.
- Add support for RSS input via new
- Instagram (scraping):
- Handle media items with no
user
object, add new fetch for comments. - Add
Instagram.merge_scraped_comments()
.
- Handle media items with no
- ActivityStreams 2:
- Handle error when
type
isn't a string.
- Handle error when
- Reddit:
- Implement
get_activities()
to fetch posts by the current user or a user specified withuser_id
.
- Implement
- Facebook scraping:
- Skip "Suggested for you" posts.
- Add
log_html
kwarg toget_activities
; defaults to False. - Miscellaneous bug fixes.
- JSONFeed:
- Handle malformed
items.author
element.
- Handle malformed
v3.2:
Notable changes
Source.original_post_discovery
: add newinclude_reserved_hosts
kwarg, defaults toTrue
.- Facebook:
- Scraping: handle pictures, videos, link attachments, and text links in timeline/news feed posts.
- Mastodon:
- Bug fix for
get_activities()
withfetch_mentions=True
: handle notifications withstatus: null
. Maybe happens when a status is deleted? create
/preview_create
: support bookmarks. (Nothing special happens with them; theircontent
is posted as a normal toot.)
- Bug fix for
- microformats2:
- Stop rendering
image.displayName
as visible text in HTML, since it's already in the<img>
'salt
attribute. - Add
bookmark-of
support. - Add
prefix_image_urls()
function. - Handle null
content
in AS1/2 objects. json_to_object
bug fix for compositebookmark-of
properties.
- Stop rendering
- Twitter:
create
/preview
: support large videos via async upload. We now passmedia_category=tweet_video
to the chunked uploadINIT
stage, and then make blockingSTATUS
calls until the video is finished processing. (bridgy#1043)create
/preview
: allow bookmarks. (bridgy#1045)create
/preview
: allow non-Twitter replies, ie activities that includeinReplyTo
URLs even if none of them point to a tweet. (bridgy#1063)get_activities
: support list ids as well as slugs.- Bug fixes for removing t.co links to quoted tweets.
- Bug fix for multiple instances of the same link in tweet text.
get_activities()
: raiseValueError
on invaliduser_id
.
- REST API: ported web framework from webapp2 to Flask. No user-visible behavior change expected.
v3.1
Notable changes
- Add Python 3.8 support, drop 3.3 and 3.4. Python 3.5 is now the minimum required version.
- Add Pixelfed! Heavily based on Mastodon.
- Standardize Instagram's and Facebook's scraping into new common
scraped_to_activities()
,scraped_to_activity()
, andmerge_scraped_reactions()
methods. - Atom:
- Add the
summary
element (#157).
- Add the
- REST API:
- Bug fix: URL-encode Unicode characters in
Link
HTTP headers (egrel=self
,rel=header
).
- Bug fix: URL-encode Unicode characters in
- Facebook:
- Scraping now uses mbasic.facebook.com instead of m.facebook.com.
- Flickr:
- Add support for adding tags to existing photos (bridgy#857).
get_comment()
: skip fetching comments from API ifactivity
kwarg is provided and contains the requested comment.
- GitHub:
- Handle HTTP 451 Unavailable for Legal Reasons responses (eg for DMCA takedowns) gracefully.
- Add create/preview support for reactions on pull review request comments (ie URLs with
#discussion_r...
fragments).
- HTML/microformats2:
- Add
aria-hidden="true"
to empty links (bridgy#947). - Bug fix: escape
&
,<
, and>
characters in bare mf2content
properties (aaronpk/XRay#102). json_to_object()
: convertnickname
tousername
.
- Add
- JSON Feed:
- Gracefully handle when
content_html
andcontent_text
are incorrectly lists instead of strings.
- Gracefully handle when
- Instagram:
- Include threaded (ie nested) comments in scraping (bridgy#958).
- Mastodon:
- Bug fix for alt text with image attachments (bridgy#975).
- Omit empty
limit
param for compatibility with Pleroma (bridgy#977).
- Meetup:
create()
: handle API errors and return the error message in theCreationResult
(bridgy#921).
- Twitter:
- Bug fix: URL-encode list names in API calls.
- Bug fix: propagate alt text into AS1
photo.displayName
so that it gets all the way into microformats2 JSON and HTML (#183).
- Reddit:
- Implement
post_id()
. - Cache user data fetched from the API for 5m to avoid repeating user profile API requests (bridgy#1021).
when fetching multiple comments or posts from the same author - Bug fix: use 'displayName' instead of 'name' in AS1 objects for submissions.
- Bug fix: use tag URIs for activity ids.
- Implement
- ActivityStreams 2:
to_as1()
: forCreate
activities, include the activity actor's data in the object's author (snarfed/bridgy-fed#75).to_as1()
: convertpreferredUsername
tousername
.from_as1()
: convertusername
topreferredUsername
.from_as1()
: bug fix, makecontext
kwarg actually work.
v3.0
Breaking changes:
- Python 2 is no longer supported! Including the App Engine Standard Python 2 runtime. On the plus side, the Python 3 runtime is now supported! See this list of differences for more details.
Non-breaking changes:
- Migrate demo app and API to the App Engine Standard Python 3 runtime.
- Instagram:
- Scraping: fetch 50 likes instead of 24. (snarfed/bridgy#898)
- Scraping bug fix for
get_actor()
withuser_id
.
- Twitter:
- Add image alt text support to
get_activites()
etc (#183).
- Add image alt text support to
- RSS:
- Add
itunes:image
,itunes:author
, anditunes:category
. - Strip HTML from
title
element (#177). Background. - Always include author in items (#177).
- Bug fix: extract feed image from
hfeed
correctly. - Bug fix: don't crash on
article
ormention
tags in items with enclosures.
- Add
- Atom:
- Bug fix: extract feed image from
hfeed
correctly.
- Bug fix: extract feed image from
- REST API:
- Add HTTP
HEAD
support. - Add support for URL fragments with
input=html
. If a fragment is provided, only that specific element is extracted and converted. (#185)
- Add HTTP
- GitHub:
- Publish: preserve
<code>
tags instead of converting them to `s so that GitHub renders HTML entities like>
inside them instead of leaving them escaped. Background.
- Publish: preserve
- JSON Feed:
- Handle malformed attachments better.
- microformats2:
- Don't crash on string
context
fields. html_to_activities()
: limit toh-entry
,h-event
, andh-cite
items (#192).
- Don't crash on string
- The
cache
kwarg toSource.original_post_discovery()
now has no effect.webutil.util.follow_redirects()
has its own built in caching now. - Added Meetup.com support for publishing RSVPs.
v2.2:
Notable changes
- Add Mastodon support!
- Add Python 3.7 support, and improve overall Python 3 compatibility.
- Update a number of dependencies.
- Switch from Python's built in
json
module toujson
to speed up JSON parsing and encoding. - Add
duration
andsize
support to ActivityStreams 1 and 2, RSS, and microformats2 HTML and JSON. microformats2 support is still emerging for both. Both integer seconds and ISO 8601 string durations are supported forduration
. Integer bytes is used forsize
everywhere. microformats2 HTML also includes human-readable strings, eg5.1 MB
. (#169) - Twitter:
[preview]_create()
: detect attempts to upload images over 5MB and return an error.
- Facebook:
- Add
get_activities(scrape=True)
for scraping HTML from m.facebook.com. Requiresc_user
andxs
cookies from a logged in session (snarfed/bridgy#886). - Upgrade Graph API version from 2.10 to 4.0.
- Add
- Atom:
- Bug fix for de-duping images in attachments.
- RSS:
- Wrap all
<description>
element contents inCDATA
sections. - Render images in
<description>
with HTML<img>
tags (#175). from_activities()
bug fix: don't crash when converting multiple attachments to enclosures in a single item. (RSS only supports one enclosure per item, so we now only include the first, and log a warning if the activity has more.)
- Wrap all
v2.1:
Notable changes
- Convert AS2
Mention
tags to AS1objectType
mention
(non-standard) and vice versa (snarfed/bridgy-fed#46). - Twitter:
- Bug fix for large block list fetches that get rate limited after a few successful requests.
- Handle HTTP 403 + error code 200 when fetching retweets for a protected or otherwise unavailable tweet (bridgy#688).
- Demote @-mentions from person-tags to mentions. Specifically, this means they'll no longer get rendered with
u-category
mf2.
- Instagram:
- Disabled in the REST API entirely due to Instagram's aggressive rate limiting and blocking (bridgy#655).
- Update scraping to handle replies in new
edge_media_to_parent_comment
field (#164). - Use cookie for all scraping HTTP requests, not just for likes.
- microformats2:
- Revise whitespace handling; use
white-space: pre
CSS in HTML output.
- Revise whitespace handling; use
- Facebook:
- Bug fix: don't interpret
photo.php
as username in post URLs.
- Bug fix: don't interpret
- Atom:
- Switch from
white-space: pre
CSS back to converting newlines to<br>
s because some feed readers (eg NewsBlur) follow it too strictly and don't even line wrap.
- Switch from
- RSS:
- Default title to ellipsized content.