consider switching from Matrix to Zulip

[pdurbin]

Part of why I supported the move from IRC to Matrix is that Matrix had bought Gitter, which had been the poster child transparent communication (especially chat) within an open source project.

However, Gitter is now private by default, while it used to be public by default:

• Gitter was public by default but now it is private by default pdurbin/slopi-communication#35

Now, that's not to say that Gitter is bad. It's just that Gitter is no longer special. Gitter is just a Matrix home server now. Matrix, like every other chat tool that I know of, is private by default.

I also was hoping (was expecting, really), that Matrix would get the equivalent of Gitter's awesome public logs. According to the Gitter has fully migrated to Matrix announcement, these are still coming in the form of Matrix Public Archive. I hope so. Meanwhile we're sort of stuck with https://view.matrix.org/alias/%23dataverse:matrix.org which is a bit buggy (often not showing today's logs) and not fully featured (can't easily link to specific timestamps).

Yesterday I did stand up Matrix Public Archive at https://people.dataverse.org/chatter/r/dataverse:matrix.org but it seems a bit buggy. The way I reported it was "images broken, links to people don't work, can't navigate to a different day." Quite possibly I misconfigured it. I'm waiting to hear back.

Meanwhile I took another look around at other chat options and I think the new poster child for transparent chat in open source project is Zulip. It's used by Rust, for example, at https://rust-lang.zulipchat.com . Chat logs for the Zulip project itself can be found at https://chat.zulip.org . One thought I have is to set up an org for Dataverse on Zulip with the public access option enabled and see if there's interest in bridging it to Matrix. That way, we get nice public logs going forward (and lots of other features) but we can continue using Matrix if we like.

Coming from IRC and Matrix, Zulip will take a little getting used to. We've always had a single #dataverse channel (or room) and people just type away in there. Zulip is based on topics, which you could think of as a thread. You name the topic and then have the conversation in there. (Matrix also has threads now, by the way, but I'm been uncertain about using them.)

A nice feature of Zulip is that with an org we can have as many "streams" as we like. Streams are like channels. So we could have channels such as:

• general
• random
• containers

Whatever we want. I think this would be much better than the community Slack, which only has 90 days of history:

• dataversecommunity.slack.com has only 90 days of history dataverse#8712

Thoughts? I'm happy to talk about it here or at https://chat.dataverse.org or wherever! 😄

[pdurbin]

The first comment was getting long but another topic is data liberation of Zulip data. (I still haven't figured this out for Matrix.)

It looks like there's a nice tool for backup up public Zulip data to HTML (and JSON): "zulip-archive works by downloading Zulip message history via the API, storing it in JSON files, maintaining its local archive with incremental updates, and turning those JSON files into the HTML archive." - https://github.com/zulip/zulip-archive

From the GUI, admins of the org can do a manual backup of public data: https://zulip.com/help/export-your-organization

Finally, if you run Zulip yourself (in which case you own the data already), there are various backup tools: https://zulip.readthedocs.io/en/latest/production/export-and-import.html

[ara4n]

project lead for Matrix here: i'm a bit surprised to hear Matrix described as a closed chat system. Public rooms in Matrix are by definition open, and with guest access enabled are indexable publicly. You're right that view.matrix.org is old and about to be replaced by the new matrix static archive service (which is still beta), but it should provide precisely the same functionality as Gitter's public archive did. This why we wrote it. It's even written by one of the Gitter developers!

So, what is it that makes you say that Matrix is closed?

[pdurbin]

@ara4n hi! I've watched a lot of your talks and Matrix Live. I'm honored that you took the time to stop by and read what I have to say! 🎉

I was just now reviewing a little doc called Dataverse Community Chat Requirements I created when we were thinking hard about moving from IRC to... something! I just added a sheet for Matrix (before I saw your comment) to try to help get my head around what I'm so wound up about, why I'm even thinking about alternatives to Matrix. The spreadsheet is half baked so please take it with a grain of salt but maybe it'll give a little insight into where my head's at.

For some context, I'm really into transparency in open source. You can read my blog post about this here: http://blog.greptilian.com/2020/01/25/slopi-communication/

I also wrote an article here that at least mentions IRC logging: https://opensource.com/open-organization/17/11/transparency-dataverse-project

Please correct me if I'm wrong but did I call Matrix closed or a closed system? It is not. I am eagerly anticipating Matrix Public Archive. As I said above, I stood it up over the weekend. I also made a couple (very small) contributions:

• Link to demo videos #150 matrix-org/matrix-viewer#153
• Switch /timestamp_to_event endpoint to stable v1 #142 matrix-org/matrix-viewer#154

https://view.matrix.org is unfortunately pretty buggy. A good amount of the time when I go to https://view.matrix.org/alias/%23dataverse:matrix.org I land on some random day instead of today, sometimes months in the past. This can leave potential members of our community with the impression that nothing has been discussed for months, which is not true at all. Some days it's hard to keep up! 😅

I haven't bothered to open an issue in matrix-static because I know it's being replaced (by Matrix Public Archive). I'm trying (and failing) to be patient. I miss the reliability of my old IRC logger. 😄

I don't actually have the impression that Matrix is indexed particularly well. If I google for "dataverse vagrant" (just picking an older technology at random) I find an old conversation captured by my IRC logger at http://irclog.iq.harvard.edu/dataverse/2020-11-09/ . And over the years of that IRC logger I would often find our logs in google searches. However, I've never found anything from #dataverse on Matrix from a google search, and we've been on Matrix since June 2021.

Matrix Public Archive should fix this indexing problem. Again, I'm really looking forward to it. 🚀

I could go on and on and I appreciate it if you've made it this far. Another thing Gitter and now Zulip provide is a wonderful preview of messages and activity when you first click, for example, https://rust-lang.zulipchat.com . I'll put a screenshot below. This screams "we care about transparency!" Before signing up or even thinking about what client to use, you can do a lot of exploring. And you can get a web link to goes exactly to the line that you're reading. For example https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/github.20bot/near/328977306 . With Matrix you get a matrix.to link that assumes you already have a Matrix client and are already signed into it. Non-Matrix people have hoops to jump through before they can read anything. This should be fixed by Matrix Public Archive. 🚀

Ok, before I write much more, and I'm sorry this is so long, I'd like to say that I really appreciate the free use of the matrix.org home server for the past year and a half. Freenode fell apart and we needed to jump ship. Overall, Matrix has been a wonderful new home for Dataverse chat. I'm just trying to address and talk through some challenges we're facing. Thank you again ever so much for listening! ❤️ ❤️ ❤️

[pdurbin]

Wow, our approval from Zulip as an open source project came through really quickly. Here's how the unauthenticated homepage looks:

The "general" stream was there by default and has a "stream events" topic that receives messages about changes to streams, such as making them public:

I see @poikilotherm kicked off a "hello I'm new here" topic, which looks good:

I think I'm happy enough with the "general" stream the way it is. I'll definitely introduce myself in the "I'm new here" topic.

[pdurbin]

I mentioned above that "stream events" show up in the "general" stream. In addition, when someone joins, a message goes to the "signups" stream in the "core team" topic:

Here's where both of these announcements are configured:

I'm prefer to have as few non-web-public streams (indicated by the lock 🔒 ) as possible. If we redirect new user announcements to the "general" stream, we could delete the "core team" stream. The only other topic under the "core team" stream is "private streams" and it's on for demonstration purposes (so safe to delete the entire "core team" stream, I'd say):

[pdurbin]

@ara4n I'm not sure if you saw my reply and high praise for your work above... #22 (comment)

On a related note, I just left the following comment:

• Deep link date in room element-hq/element-web#24914 (comment)

[pdurbin]

I'm prefer to have as few non-web-public streams (indicated by the lock 🔒 ) as possible. If we redirect new user announcements to the "general" stream, we could delete the "core team" stream. The only other topic under the "core team" stream is "private streams" and it's on for demonstration purposes (so safe to delete the entire "core team" stream, I'd say):

Just a quick update on streams:

• We did delete (archive) the #core team stream that came out of the box: https://dataverse.zulipchat.com/#narrow/stream/375876-zulip/topic/.E2.9C.94.20out.20of.20the.20box.20.22core.20team.22.20stream
• There is light discussion of having a private #security stream: https://dataverse.zulipchat.com/#narrow/stream/375876-zulip/topic/private.20channel.20for.20security.3F
• We've been creating public streams here and there. Please see the screenshot below.

[pdurbin]

Oh, and if anyone has feedback on the Zulip experience itself at https://dataverse.zulipchat.com, please use the #zulip stream. Here's a look at recent topics:

[pdurbin]

You're right that view.matrix.org is old and about to be replaced by the new matrix static archive service

@ara4n any update on this? https://archive.matrix.org/r/dataverse:matrix.org was working great for a while but now the whole service is on hold:

I also asked about this in the Matrix Viewer room: