Opening up development of Weave Net

[dholbach]

To those of you who responded in the discussion of #3939, thanks a lot for your ideas and offers of help so far! 💖

Weaveworks wants to open up the development of Weave Net and in order to bring interested folks together, get to know each other and discuss ideas, we would like to arrange a meeting soon. 📆

If you are interested in joining, please comment on this issue, so we can let you know when it's going to happen, or give you a ping for a doodle we've set up. Thanks a lot everyone in advance!

CCing @monadic and @kingdonb

[rajibmitra]

@dholbach - I would like to join, please add me to the meeting.

[nathanejohnson]

I'm very interested in this. Thank you!

[rajch]

I am interested. Thank you.

[monadic]

Hi folks, we'll target in a meeting in the last week of JUNE 2022. I'll choose a time for the community meeting, which will likely be around 7am Pacific / 3pm UK.

[yrsurya]

Interested and also like to know the outcome of meetings that happened

[monadic]

Hi would folks be able to meet next week on 10 AUGUST at 4pm UK time?

[jscottsf]

Same here. Interested.

[monadic]

Great -- here is the Meeting URL to join

Weave Net community meeting
Wednesday, August 10 · 4:00 – 4:40pm UK (8am US Pacific)

Google Meet joining info
Video call link: https://meet.google.com/ydc-wwtn-xye
Or dial: ‪(GB) +44 20 3910 5019‬ PIN: ‪732 425 366‬#
More phone numbers: https://tel.meet/ydc-wwtn-xye?pin=1413075876243

[monadic]

Next meeting 31 August!

weave net open meeting
Wednesday, August 31 · 4:00 – 4:50pm
Google Meet joining info
Video call link: https://meet.google.com/fjy-msnk-fcq
Or dial: ‪(GB) +44 20 3956 9319‬ PIN: ‪208 155 944‬#
More phone numbers: https://tel.meet/fjy-msnk-fcq?pin=2035335265953

[monadic]

[monadic]

^ millions of users

[monadic]

So the next steps:

• we have a core group of 3 people who want to move the community forward
• we want to get this to 10 or more
• weaveworks will support as much as possible - our only asks are "goes into CNCF" and we are all good community members!

-alexis

[monadic]

comments welcome people

[rajch]

Which of the following do we need most at this point?

1. Regular maintenance, release publishing, documentation updates, pull request curation, regressions against newer library versions etc.
2. Further development, feature additions etc.
3. Creation of new processes and structures to make the project more suitable for community participation and CNCF inclusion.

I think that, at this point, (1) is most important - just to keep weave net where it is. Maybe interested community members can begin by sharing some of that responsibility.

Sorry I missed the meeting. Mistimed it by one hour because of a silly timezone mistake.

[richardcase]

Would love to also help with this if it isn't too late 🤞

[jmickey]

Count me in as well!

[monadic]

Which of the following do we need most at this point?

1. Regular maintenance, release publishing, documentation updates, pull request curation, regressions against newer library versions etc.
2. Further development, feature additions etc.
3. Creation of new processes and structures to make the project more suitable for community participation and CNCF inclusion.

I think that, at this point, (1) is most important - just to keep weave net where it is. Maybe interested community members can begin by sharing some of that responsibility.

Sorry I missed the meeting. Mistimed it by one hour because of a silly timezone mistake.

1 is the most important!

[jscottsf]

I would agree (1). Triage on issues, fix high priority ones, also update dependencies.

[richardcase]

I think the order listed is spot on.

[richardcase]

Wednesday, August 31 · 4:00 – 4:50pm

@monadic - Is this BST?

[monadic]

Yes

[monadic]

thank you to @rajch @nathanejohnson and others for help and enthusiasm so far

this is the PLAN

1. we shall make a community document that states the goals for the next 6-12 months
2. priority is maintenance and regaining momentum as a self-sustaining community
3. we shall have a monthly open community meeting online to discuss and vote on a PUBLIC ROADMAP which will lay out near term priorities, how to help etc
4. step one is that I shall work on this with @rajch @nathanejohnson who will post link below so that others can help
5. overall aim is to get to X new maintainers by time (some time in 2023) to be determined
6. governance will aim to cede more control as trust builds up until the project is governed by a strong community
7. ex maintainers have offered to help with this process

next public meeting will be 28 Sep at 4pm UK // 8am Pacific

--alexis

[rajch]

I have written up a basic outline for the roadmap here. Just broad headings, with a few points that I thought of. If the structure is okay, I shall continue to fill it in.

The Google doc is public editable. Feel free to jump in.

[monadic]

added some text!

[rajch]

Some things that we can do/should do with the aim of CNCF inclusion:

1. Start gathering a list of adopters (people or projects that are using weave net right now). See here for a definition of "adopter".
2. Consider the CNCF IP Policy.

Full details about the graduation process can be found here.

[fujitatomoya]

count me in 👍

[rajch]

I've been disconnected for a bit. Did the public meeting on 28th September happen? What did I miss?

[rajch]

Proposal for new install endpoint

The recommended method for installing weave used to be this:

kubever=$(kubectl version | base64 | tr -d '\n')
kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=$kubever

This would redirect to the simpler url:

https://cloud.weave.works/k8s/vX.YZ/net.yaml

which would generate a manifest appropriate to the version of kubernetes. In the current release of weave (2.8.1), there are four possible manifests: weave-daemonset-k8s-1.8.yaml, weave-daemonset-k8s-1.9.yaml, weave-daemonset-k8s-1.11.yaml, and weave-daemonset-k8s.yaml. The last one was added recently, and the documentation was updated to use that from the release, like this:

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

I propose that the community create and maintain an endpoint like the old https://cloud.weave.works/k8s one, to ensure that the endpoint remains the same for potential future releases of both weave and kubernetes. This could re-use the old weave cloud code, or be a fresh implementation.

I can take this up if other people think this is a good idea.

[richardcase]

Is anyone going to Kubecon NA? Perhaps we could meetup if there are a few of us going?

[monadic]

@rajch ?

[rajch]

Sadly, not going. 😞

[rajch]

Proposal for new install endpoint

So I set up an experimental install endpoint, which works with both URL patterns mentioned above. I've tried to be as faithful to the old https://cloud.weave.works/k8s endpoint behaviour as I could remember. The new endpoint is currently hosted on a Free-tier Azure Web App, accessible at https://weave-community-downloader.azurewebsites.net/.

Weave can now be installed on a kubernetes cluster using:

kubever=$(kubectl version | base64 | tr -d '\n')
kubectl apply -f https://weave-community-downloader.azurewebsites.net/k8s/net?k8s-version=$kubever

OR

kubectl apply -f https://weave-community-downloader.azurewebsites.net/k8s/v1.25/net.yaml

where the v1.25 part can be replaced with any kubernetes version down to 1.8.

Source code here: https://github.com/rajch/weave-endpoint

If it's good enough, maybe we could find a friendlier URL for it 😁

[ledzepppelin]

what about other options? like setting MTU or disabling NPC? will they work on new endpoint?

[rajch]

I need some help implementing those. I have the list of options, but don't remember what all of them did. If anyone has any old generated YAML files, looking at those would help. I'll implement the ones I do remember, and post progress here.

[rajch]

Some options have now been implemented: specifically, environment variables (MTU can be set using this) and SELinux options. Like I said, I need a bit of help for the others, like disabling NPC. If anyone remembers the old behaviour, or has the generated yaml, it would help. Meanwhile, I will try to reverse engineer using the official weave-kube image.

See the GitHub README for details.

[ledzepppelin]

this one is pretty old though @rajch

apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: weave-net
      annotations:
        cloud.weave.works/launcher-info: |-
          {
            "original-request": {
              "url": "/k8s/net?k8s-version=1.16&disable-npc=true",
              "date": "Wed Sep 09 2020 13:33:36 GMT+0000 (UTC)"
            },
            "email-address": "support@weave.works"
          }
      labels:
        name: weave-net
      namespace: kube-system
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: weave-net
      annotations:
        cloud.weave.works/launcher-info: |-
          {
            "original-request": {
              "url": "/k8s/net?k8s-version=1.16&disable-npc=true",
              "date": "Wed Sep 09 2020 13:33:36 GMT+0000 (UTC)"
            },
            "email-address": "support@weave.works"
          }
      labels:
        name: weave-net
    rules:
      - apiGroups:
          - ''
        resources:
          - pods
          - namespaces
          - nodes
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - networking.k8s.io
        resources:
          - networkpolicies
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ''
        resources:
          - nodes/status
        verbs:
          - patch
          - update
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: weave-net
      annotations:
        cloud.weave.works/launcher-info: |-
          {
            "original-request": {
              "url": "/k8s/net?k8s-version=1.16&disable-npc=true",
              "date": "Wed Sep 09 2020 13:33:36 GMT+0000 (UTC)"
            },
            "email-address": "support@weave.works"
          }
      labels:
        name: weave-net
    roleRef:
      kind: ClusterRole
      name: weave-net
      apiGroup: rbac.authorization.k8s.io
    subjects:
      - kind: ServiceAccount
        name: weave-net
        namespace: kube-system
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: weave-net
      annotations:
        cloud.weave.works/launcher-info: |-
          {
            "original-request": {
              "url": "/k8s/net?k8s-version=1.16&disable-npc=true",
              "date": "Wed Sep 09 2020 13:33:36 GMT+0000 (UTC)"
            },
            "email-address": "support@weave.works"
          }
      labels:
        name: weave-net
      namespace: kube-system
    rules:
      - apiGroups:
          - ''
        resourceNames:
          - weave-net
        resources:
          - configmaps
        verbs:
          - get
          - update
      - apiGroups:
          - ''
        resources:
          - configmaps
        verbs:
          - create
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: weave-net
      annotations:
        cloud.weave.works/launcher-info: |-
          {
            "original-request": {
              "url": "/k8s/net?k8s-version=1.16&disable-npc=true",
              "date": "Wed Sep 09 2020 13:33:36 GMT+0000 (UTC)"
            },
            "email-address": "support@weave.works"
          }
      labels:
        name: weave-net
      namespace: kube-system
    roleRef:
      kind: Role
      name: weave-net
      apiGroup: rbac.authorization.k8s.io
    subjects:
      - kind: ServiceAccount
        name: weave-net
        namespace: kube-system
  - apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: weave-net
      annotations:
        cloud.weave.works/launcher-info: |-
          {
            "original-request": {
              "url": "/k8s/net?k8s-version=1.16&disable-npc=true",
              "date": "Wed Sep 09 2020 13:33:36 GMT+0000 (UTC)"
            },
            "email-address": "support@weave.works"
          }
      labels:
        name: weave-net
      namespace: kube-system
    spec:
      minReadySeconds: 5
      selector:
        matchLabels:
          name: weave-net
      template:
        metadata:
          labels:
            name: weave-net
        spec:
          containers:
            - name: weave
              command:
                - /home/weave/launch.sh
              env:
                - name: HOSTNAME
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: spec.nodeName
                - name: EXPECT_NPC
                  value: '0'
              image: 'docker.io/weaveworks/weave-kube:2.7.0'
              readinessProbe:
                httpGet:
                  host: 127.0.0.1
                  path: /status
                  port: 6784
              resources:
                requests:
                  cpu: 50m
                  memory: 100Mi
              securityContext:
                privileged: true
              volumeMounts:
                - name: weavedb
                  mountPath: /weavedb
                - name: cni-bin
                  mountPath: /host/opt
                - name: cni-bin2
                  mountPath: /host/home
                - name: cni-conf
                  mountPath: /host/etc
                - name: dbus
                  mountPath: /host/var/lib/dbus
                - name: lib-modules
                  mountPath: /lib/modules
                - name: xtables-lock
                  mountPath: /run/xtables.lock
          dnsPolicy: ClusterFirstWithHostNet
          hostNetwork: true
          hostPID: true
          priorityClassName: system-node-critical
          restartPolicy: Always
          securityContext:
            seLinuxOptions: {}
          serviceAccountName: weave-net
          tolerations:
            - effect: NoSchedule
              operator: Exists
            - effect: NoExecute
              operator: Exists
          volumes:
            - name: weavedb
              hostPath:
                path: /var/lib/weave
            - name: cni-bin
              hostPath:
                path: /opt
            - name: cni-bin2
              hostPath:
                path: /home
            - name: cni-conf
              hostPath:
                path: /etc
            - name: dbus
              hostPath:
                path: /var/lib/dbus
            - name: lib-modules
              hostPath:
                path: /lib/modules
            - name: xtables-lock
              hostPath:
                path: /run/xtables.lock
                type: FileOrCreate
      updateStrategy:
        type: RollingUpdate

[rajch]

Thanks! That gave me what I needed to implement the disable-npc query parameter. That's live now.

We still need to implement known-peers and trusted-subnets. I guess they just set the corresponding environment variables.

Lastly, there's password-secret. I have never used it, so don't know what exactly is supposed to happen.

Anyone know about these?

[rajch]

Happy 2023, everyone. Any action on this front?

[kingdonb]

@rajch I am trying to ascertain who is the person who can best handle an issue that appears to need attention from the new release team. I have not been to any of the meetings scheduled so I am not sure who that is yet. I think we need some of us people to speak up, say who are the new maintainers, and create a new website with that information on it and some other details.

There's a single comment on this issue, comment from (a prominent community member) that explains how we can resolve
this important issue with latest in the release YAML.

Ref::
#3960 (comment)

There is work happening to improve the maintenance of Weave Net and eventually donate it somewhere, likely CNCF, but it's a slow process. I'm not the one doing this work. I'm not sure who are the ones doing that work, or if they're still on holiday as for some people it's surely that time of year as well, so volunteer obligations can take a backseat. That's understandable.

I'm actually suggesting that whomever is driving this effort can contact me, I'll help with hosting. We can build websites.

If it's not clear why I think we need a new website as a next step right now, please check the other thread that I linked above for follow-up on that discussion. I want to keep this topic focused and respectful, since esp. volunteers who have yet to be identified are quite easily chased away.

[rajch]

In the absence of any other forum, I hereby and herein 😃 announce that the alternate install endpoint at https://weave-community-downloader.azurewebsites.net/k8s/ has handled the issue mentioned by @kingdonb above.

[kingdonb]

(Thanks for this!) Progress 🚀 !

[rajch]

Sorry, the whole URL is https://weave-community-downloader.azurewebsites.net/k8s/v1.25/net.yaml

[kingdonb]

After our meeting today, as a bit of status report, I think that the main takeaway was we all still find value in Weave Net and want to see it go on being useful into the future, but there is no maintainer.

And it's clear that recommending its use now is very tenuous, with the maintenance situation (there is no current maintainer, and there have been no releases in over a year.) We'd like to resolve that, but also, nobody from the current group actively wants to sign up to be "lead maintainer" and we're going to need at least one of those.

Release goals: It is a priority to make a new release of the docs, that clarifies the multi-arch issue from #3974 and the maintenance situation as we've been discussing in #3960. There are other priorities, but until a new distrib endpoint can at least resolve these issues it's not worthwhile spending time enumerating them all here.

To be clear, there hasn't been a maintainer step up yet, but we have folks that are interested in providing mentorship or guidance for volunteers that might be interested in that role.

---

So, we're quite a few steps earlier than "Weave Net Foundation" but hopeful that like other great Open Source projects from Weaveworks that grew legs and went on to new lives, maybe this one still can too. We likely have the expertise in the current group to produce at least one more new release, but it is a question of time (do we all want to spend the time required to validate a new release together) and also a question of whether we have the interest to do much more than that.

And another question whether: if we need to create a new organization in order to host the revived Weave Net, or can it still be salvaged in-place without an actual fork. I think that Weaveworks will not want to be seen as "behind" the new releases.

I'm in favor of forking at this point, but I won't be the one to throw the switch on a "new org", or give it a name, (or buy a domain name, etc.) and there's no reason to rush these things at all; we can all work things out in our own personal forks. Looking forward to continuing this conversation here over time. Thanks for the great meeting today!

[fujitatomoya]

@kingdonb @rajch

as i mentioned, that i can help run the weave on raspi4 arm64 platform with kubernetes. but i am really new here to understand the procedure.

like,

• We need to build the image by ourself? if so, any configuration or procedure to do so?
• What kind of release test do we need to run to confirm the image? Is there any test integrated in repository?

more questions probably come up for me, but those are my 1st questions here.

[rajch]

@kingdonb @fujitatomoya

Thanks for a great meeting.

Let's try to produce at least one more release, as a first step. This exercise will, I believe, show us whether non-weaveworks-led maintenance is even feasible. I suggest we do this as an informal exercise, for "internal" consumption, without forking or setting up a new org.

Here is a list of tasks. Feel free to add/edit.

1. Learn and use the existing build and test infrastructure. In the process, perhaps we can make the dependency inventory, and perhaps begin to evolve and document a new infrastructure. I volunteer to take point on this one.
2. Make a new release of the docs, highlighting current issues and known workarounds. This is possibly a bigger effort than (1) above.
3. Create a new distrib endpoint which uses/monkey-patches the current release to solve at least some issues. I took a stab at this: details here and here. The source is here. It would be nice if you guys could also test it.

Apropos point (1) above and also the first question asked by @fujitatomoya, there is some documentation on the weaveworks website for building weave net, here, but it looks outdated. There are also tests in the repository itself, here. I'll be spelunking in that stuff, and will report any interesting findings here.

[rajch]

Something else that occured to me afterwards: perhaps we could create a test suite for only the CNI plugin aspect of weave net . This would test kubernetes and application functionality on top of weave net, such as user-defined pod and service CIDRs, session affinity, multicast, network policies, iptables vs nftables, different versions of CNI, containerd, different versions of kubernetes, different processor architectures, etc.

This can be an independent activity, not directly related to any of the tasks in my earlier comment. It would provide a "higher" layer of testing, that can be applied to the existing builds, as well as to any new builds we eventually create. Given a suitably diverse test infrastructure, it would catch many of the kind of issues we have been seeing lately.

I'm thinking of something that can be run immediately after creating a new kubernetes cluster via kubeadm and applying the weave net add-on. Something like the Kubernetes E2E tests.

[kingdonb]

The existing tests I was able to identify right away, that we will need to replicate in our new CI infrastructure, were previously hosted on CircleCI:

• https://github.com/weaveworks/weave/blob/master/.circleci/config.yml

I believe we can safely assume that any CircleCI credentials have been disabled at this point, and are not available to use.

The tests in there are three-fold: unit tests, smoke tests, and code coverage analysis.

I think apart from playing back all the ideas represented above for myself, building a personal release and testing the new/existing published artifacts out locally, I'll be working out how to present code coverage for internal consumption, so that team members can begin to understand the parts of the code that are tested and see also what code the tests maybe don't cover.

I'm strongly in support of the "new e2e tests" idea, if we can encapsulate the full release process and verify those artifacts in a test pipeline that does at least as much as before, plus our new e2e, then I think it should be much easier to publish new releases in the future on a schedule. That would give me enough to validate and metric that I can put a "seal of approval" on our products, so that Weaveworks can pass the torch.

[kingdonb]

Believe it or not, this week is the first Thursday of March!

Who is still interested, should we plan to meet? I can create a calendar invite and have a paid Zoom account so we can host the meeting.

I have not built the weave net artifacts as we discussed we would all try to do, or run them through end-to-end tests yet myself, but I have high hopes that I will still get around to it before Thursday. I've been focused on some weird issues ranging from containerd upgrades to backwards compatibility for Docker engine in one of my other open source projects.

I'm happy to ring the bell and call the meeting, but if everyone else has made as little progress as I have made on Weave Net, then it's going to be a short meeting... schedule/postpone/cancel?

Please add your $0.02 if you are interested in attending the next meeting, (or e-mail kingdon at weave dot works and I'll be sure you get included on the Zoom invite.) Thanks everyone for your interest in Weave Net!

[fujitatomoya]

@kingdonb unfortunately i could not allocate time for this, but happy to join if the meeting takes place.

[rajch]

I'm in the same boat, brothers. Day job took up too much time.

I propose we postpone by a week, and see if we can put in a little work in this time. That way, the meeting can be a bit longer :-). So, 9th March?

[kingdonb]

March 9 works better for me, I'll put together a calendar invite!

You are both on the list ✅

As anyone else has interest in following the development or participating as a maintainer, please do note your interest here!

[kingdonb]

FWIW, we have met on March 9 and we have scheduled another meeting for April 13

There has been progress in an e-mail thread, not that we can report back to the whole community yet, but I'm really pleased to see some work has gone into the aarch64 support again, and we have reached at least a version of it building and working on Raspberry Pi devices again, albeit with some issues, (that's something we've seen was missing from the last build that Weaveworks published – the arm64 support – there is a new build scaffold, and it replaces some of the dated scaffolding.)

We are still on the lookout for volunteers, and potential future maintainers! Email kingdon at weave dot works to be added to the calendar invite for April 13. (We've settled on the second Thursday for our meeting cadence, for as long as that works!)

[fujitatomoya]

@kingdonb @rajch

It would be probably nice to have dedicated thread to debug the latest image process or debugging mostly instead of offline email communication? Or we can post debugging information here?

[rajch]

I think a dedicated thread or threads would be nice.

[fujitatomoya]

@kingdonb @rajch just created #3989. you can update the header if i got anything wrong. I will post the debug information for the verification on that issue.