Discovery and Searching Across Collective Instances

[panzerstadt]

Forgive me if it has been answered somewhere, but I can’t seem to find it, but how are we going to link these meetup.com-in-a-boxes together to find events across different interests and organizations?

An aggregator microservice?
A directory page like github /awesome?
An elasticsearch server hosted by freecodecamp powered by donations?

The reason why meetup.com succeeds is because of network effects (as mentioned in the medium article), but if each chapter is an isolated db, network effects don’t happen.

[panzerstadt]

Actually, I’d like to float an idea.

preamble

This suggestion makes every chapter’s User experience the sum of all connected chapters.
To every user using the different portals, it feels like the entire network instead of only accessing events from a single chapter (analogy: one Meetup.com instead of separate Meetup.coms you have to keep track of to find the events you want)

method

• there is a github repo/machine-readable list (like a csv or even a properly formatted readme) that all chapters can apply to be in (like /awesome repos), with their url.
• each instance of chapter (wherever it is hosted) exposes an endpoint that other instances can query for a list of events they have
• users who search will trigger queries to all running chapters that return lists of events, which the instance can process to return their most relevant results (like nearest/ soonest within the same city)

Storing user metadata

• each chapter hosts their own set of users, which store saved lists of events both within and outside the chapter.

Problems

• notifying users of event updates (are they handled by the chapters hosting events? If so, they need to store external users who have rsvp-ed their events)

[kognise]

#40 is along these lines as well, just want to link the two issues up

[cameronsr]

Referencing this, #36 and #40 , would it be possible/make sense for people to be able to volunteer to host a local server for the purposes of regional aggregation? The idea being that if part of the network goes down, there's still local backup? This could be useful for more rural areas.

[allella]

@panzerstadt would you mind changing the title to something like

Discovery and Searching Across Collective Instances

to make it easier for folks to find and to align with the vocabulary that's developing?

[panzerstadt]

@allella thanks, done that!

[mc962]

This is probably where the Elasticsearch bullet listed before would be useful (for searching at least). People could submit certain key pieces of data (name, url, image, description, etc.) to the maintainers of the Elasticsearch instance, and on indexation, you would have an easily searchable database, sort of like a Google search engine custom-built for searching for 'chapters'. Note, it doesn't have to be Elasticsearch, just a Postgres database would likely work in the short-medium term, but ES seemed to be appropriate for a component that is basically there for searching.

There could maybe be one main one that is maintained by the chapter project core devs, or something like that...although theoretically anyone could do the same if they want. This might put a lot of control/responsibility on the maintainers of this core index, but theoretically anyone could spin one up for their own purposes if they want to pay for hosting it, and the project could maybe even include the tools/instructions for that process as well.

[panzerstadt]

@mc962 i worry that having to spin up anything, es or postgres or anything, will have an associated cost that will have to be shouldered by one party (funded or not). it kinds put pressure on that party to keep it up because if it fails, all chapters fail, which is worrying.

Gun, suggested at #40 sounds like a very interesting way to solve that, every chapter just "syncs" all the time, meaning there will be a fully replicated list of searchable events on each chapter, ensuring that one failure doesn't take down others. (personally very new to the idea of federated apps, so i don't know what cons there are to that)

if there is a worry of replicas getting too large, i'm guessing naively that there's a way to replicate only 14 days worth of information, or maybe reference how notabug.io does it (also referenced at #40 , as it seems to be a globally searchable p2p reddit (replace posts with events and now you have a decentralized events list)

[allella]

For my personal use-case, I want to continue to maintain a Code For Greenville supported application that shows all tech meetup events in the Greenville, SC area. This is currently syndicating, in near real-time, Meetup.com and Eventbrite.com, with plans to support rich snippets, Facebook (very little use, so probably not), and eventually Chapter.

We will likely setup a "Greenville" Chapter organization for smaller, local meetups with no budget or national affiliation. However, I would also expect national orgs (Code For America, Women Who Code, freeCodeCamp) to host their own Chapter organizational instance where our local Greenville chapters could host their meetup events .

For this situation it's not practical to expect every Chapter organizations in the world to syndicate and promote what is happening in all the others. Rather, as long as 3rd party applications can ping each Chapter's API, assuming that information isn't marked as private by the Chapter, then we'd be able to easily create our own tools.

My question is what use cases do we see where all Chapters want to hold events for potentially millions of events all over the globe? Or, are we suggesting that any Chapter should selectively be able to sync with any other number of Chapters of its choice?

I see a case for allowing easy authentication if a user happens to sign up for multiple instances, but it feels like any aggregation of local, regional, country, industry, hobby, etc event data would be built by 3rd party apps that also have to pull in events from services like Eventbrite, Meetup.com (not everybody is going to leave) and other popular event services.

[mc962]

I also feel like Syncing across many instances isn't necessarily a trivial thing to get right.

In regards to 'spinning things up', something will need to be spun up. At the very least a database seems likely. ES also isn't necessarily that hard to spin up either, and if you integrate it with the ORM, then loading it up with the data isn't necessarily so difficult.

There's definitely value in the simplicity of simply searching with carefully crafted DB queries, and I think that's the best solution at this point, but at some point a technology dedicated to searching and returning quality results may be more worth it. That point will probably only make sense when there is enough data to need better searching...so it's not necessarily something that needs as much planning for right now, but it would be good to keep in mind when setting up search methods/endpoints (do it in a way that might support switching out the search provider). I did look at Postgresql's textsearch capabilities, and while that did look interesting, I think it has limits compared to other systems and the API looked a bit more difficult to work with (along with probably requiring more work from the database instance).

A 'common index' to me would be useful if I just wanted to find new things in the area. For example, I don't necessarily know that something like a "Code for Greenville" (to use the example chapter listed above) exists in my area, and I'd be less likely to discover it without being able to have some sort of central index that can filter on all coding chapters in my area.

[QuincyLarson]

We should keep discussing this. I've added the Roadmap label since this is something we will want to eventually build out (though not for our MVP). This is closely related to #40 so I am closing it in favor of #40.

[allella]

For those who missed it, here are the Nov 15th meeting decisions on an MVP for authentication, which is just a Google login integration to start.

Quincy put this conversation on the roadmap for after an MVP.

[strypey]

A number of projects have been experimenting with using ActivityPub to power federation between instances, including GetTogether, Mobilizon, Gancio, and Gath. There's some discussion of the nuts and bolts of this at the SocialHub forum for AP developers.