Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redesign Activity data model #286

Closed
23 of 24 tasks
snarfed opened this issue Nov 16, 2022 · 5 comments
Closed
23 of 24 tasks

Redesign Activity data model #286

snarfed opened this issue Nov 16, 2022 · 5 comments

Comments

@snarfed
Copy link
Owner

snarfed commented Nov 16, 2022 •
edited

Right now, for outbound activities, we store an Activity entity per delivered inbox. We also show per-delivered-inbox activities in the recent activities UI on user pages and https://fed.brid.gy/recent. Obviously not ideal.

Ideally we'd rethink the data model. As a short term workaround until then, we should probably also only show one row per outbound activity in the UI, not one per delivered inbox.

Before launch:

  • Expand to generic Object kind
  • Should id be key? Or prop?
    • always synthesize id if necessary
  • Store AS1 in prop, convert on write
  • populate type: verb for activity, objectType otherwise
  • populate object_ids
  • add sent, unset, failed, skipped repeated properties for send (Redesign protocol code #388) delivery
  • activity label
  • use notif/feed/user labels to generate user and feed pages
    • feed label should only be for posts/objects, not activities, so we always render the latest version
      Split current dashboard page into user and notifs?
  • triage TODOs

Activity => Object migration:

  • iterate in key order
  • collect Activitys with same source, populate ap_* fields, earliest created, latest updated
  • fill in new fields: labels, type, object_ids
  • write separate Objects for activities vs objects (notes, articles, actors)
    (this is now required by render.py)
  • print keys as we go so we can resume if necessary

After launch:
@snarfed snarfed mentioned this issue Jan 20, 2023
Closed
@snarfed snarfed changed the title Redesign data model or consolidate activities in UI Redesign Activity data model Jan 26, 2023
@snarfed snarfed mentioned this issue Jan 27, 2023
Closed
snarfed added a commit to snarfed/granary that referenced this issue Jan 28, 2023
@snarfed
test_as1: new test data used in Bridgy Fed
46d47d2 
snarfed/bridgy-fed#286
snarfed added a commit that referenced this issue Jan 28, 2023
@snarfed
Activity => Object: update xrpc_feed.py
07d24e5 
#286
snarfed added a commit that referenced this issue Jan 28, 2023
@snarfed
Activity => Object: update follow.py
7ee4e13 
#286
snarfed added a commit that referenced this issue Jan 28, 2023
@snarfed
Activity => Object: update pages.py
60eece8 
#286
snarfed added a commit that referenced this issue Jan 29, 2023
@snarfed
Activity => Object: update activitypub.py
476a9ce 
#286
snarfed added a commit that referenced this issue Jan 29, 2023
@snarfed
Activity => Object: update webmention.py
1b27e3c 
#286
snarfed added a commit that referenced this issue Jan 29, 2023
@snarfed
Activity => Object: populate ap_delivered, ap_undelivered, ap_failed
59624d1 
#286
snarfed added a commit that referenced this issue Jan 29, 2023
@snarfed
Activity => Object: populate ap_* fields for inbox delivery results
0f5034c 
#286
snarfed added a commit that referenced this issue Jan 29, 2023
@snarfed
Activity => Object: populate Object.type
b7d1e26 
#286
@snarfed snarfed mentioned this issue Jan 30, 2023
Closed
2 tasks
snarfed added a commit that referenced this issue Jan 30, 2023
@snarfed
Activity => Object: populate Object.object_ids property
18ab544 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: update xrpc_feed.py
730b003 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: update pages.py
e0cafa7 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: update activitypub.py
a88129c 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: update webmention.py
01c50cd 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: populate ap_delivered, ap_undelivered, ap_failed
4cff211 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: populate ap_* fields for inbox delivery results
7614f00 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: populate Object.type
af93255 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: populate Object.object_ids property
e436b82 
#286
snarfed added a commit that referenced this issue Jan 31, 2023
@snarfed
Activity => Object: finish ap_delivered, ap_undelivered, ap_failed
00d4dc9 
#286
snarfed added a commit that referenced this issue Feb 1, 2023
@snarfed
Activity => Object: fully populate labels, source_protocol webmention
ce2d009 
#286
snarfed added a commit that referenced this issue Feb 1, 2023
@snarfed
Activity => Object: add activity label, unify Objects for AP => wm
033ac03 
#286
snarfed added a commit that referenced this issue Feb 1, 2023
@snarfed
Activity => Object: use StructuredProperty for Object.delivered etc, …
cee6219 
…add protocol

#286

ndb implements StructuredProperty by hoisting each nested property into a
corresponding property on the parent entity, prefixed by the StructuredProperty's
name, eg delivered.uri, delivered.protocol, etc.

For repeated StructuredPropertys, the hoisted properties are all repeated on the
parent entity, and reconstructed into StructuredPropertys based on their order.

https://googleapis.dev/python/python-ndb/latest/model.html#google.cloud.ndb.model.StructuredProperty
snarfed added a commit that referenced this issue Feb 3, 2023
@snarfed
Activity => Object: use 'in progress' status
9488b40 
#286
snarfed added a commit that referenced this issue Feb 3, 2023
@snarfed
Activity => Object: add script to migrate existing Activitys to Objects
78bdfc3 
#286

temporarily adds back Activity model definition, will remove again later.
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: update xrpc_feed.py
d701c4b 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: update pages.py
3fef565 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: add script to migrate existing Activitys to Objects
c70962b 
#286

temporarily adds back Activity model definition, will remove again later.
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: update xrpc_feed.py
d655ea0 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: update pages.py
8176cd1 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: update activitypub.py
060ad96 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: update webmention.py
58e836b 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: populate ap_delivered, ap_undelivered, ap_failed
9d4e733 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: populate ap_* fields for inbox delivery results
d72be97 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: populate Object.type
eafba6d 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: populate Object.object_ids property
97ffc10 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: finish ap_delivered, ap_undelivered, ap_failed
a76fe45 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: fully populate labels, source_protocol webmention
4d2fcdd 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: add activity label, unify Objects for AP => wm
f9891b6 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: use StructuredProperty for Object.delivered etc, …
532ccb8 
…add protocol

#286

ndb implements StructuredProperty by hoisting each nested property into a
corresponding property on the parent entity, prefixed by the StructuredProperty's
name, eg delivered.uri, delivered.protocol, etc.

For repeated StructuredPropertys, the hoisted properties are all repeated on the
parent entity, and reconstructed into StructuredPropertys based on their order.

https://googleapis.dev/python/python-ndb/latest/model.html#google.cloud.ndb.model.StructuredProperty
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: use 'in progress' status
0826512 
#286
snarfed added a commit that referenced this issue Feb 5, 2023
@snarfed
Activity => Object: add script to migrate existing Activitys to Objects
2c3a693 
#286

temporarily adds back Activity model definition, will remove again later.
@snarfed
Copy link
Owner Author

snarfed commented Feb 5, 2023

I've tentatively deployed this just now and migrated all stored Activitys to Objects. Hasn't blown up yet...

@snarfed
Copy link
Owner Author

snarfed commented Feb 6, 2023

Looking ok! UI pages all seem ok, webmentions and AP inbox delivery both directions seem ok, #361 is fixed. No crashes yet. Last Activity entity was written at 8:25:54a PT this morning, Objects are flowing and getting stored ok. Suspiciously uneventful so far...but I won't complain. Merging this branch into main!

@snarfed
Copy link
Owner Author

snarfed commented Feb 6, 2023

Tentatively closing. Onward to #388...

@snarfed snarfed closed this as completed Feb 6, 2023
@snarfed
Copy link
Owner Author

snarfed commented Feb 6, 2023

Actually, let me keep this open to track all the post-launch work.

@snarfed snarfed reopened this Feb 6, 2023
This was referenced Feb 6, 2023
Closed
Closed
snarfed added a commit that referenced this issue Feb 7, 2023
@snarfed
user page activities: show counts of delivered, failed, undelivered
ca2bffd 
for #286
snarfed added a commit that referenced this issue Feb 7, 2023
@snarfed
user page activities UI: if actor has no displayName, default to user…
c143192 
…name

#286
@snarfed
Copy link
Owner Author

snarfed commented Feb 7, 2023

Converted the remaining post-launch tasks to their own issues. Closing this one. Big milestone, woo!

@snarfed snarfed closed this as completed Feb 7, 2023
snarfed added a commit that referenced this issue Feb 7, 2023
@snarfed
user page activities UI: fall back to summary for content
48252bf 
for #286
@snarfed snarfed mentioned this issue Feb 8, 2023
Closed
snarfed added a commit that referenced this issue Feb 8, 2023
@snarfed
fix delivering reposts to followers
cb66a2f 
evidently broken by #286. fixes #418
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@snarfed