Move soot to another Github organization

[ericbodden]

Currently Soot lives under the Sable organization. This was fine for a long time, but over the years several PhD students and even regular students have obtained owner access to the Sable organization - for good cause. The problem is that implicitly all these people have write access to Soot. Given Soot's now heavy industrial use, I think this should change. I have discussed this issue with Clark Verbrugge of the Sable lab and he recommended moving Soot to another organization. We could move it to "secure-software-engineering" or some new organization that we establish for that purpose, e.g. a separate "soot" organization. This might make sense, actually, so that we can also host Soot extensions there if we want.

Let's discuss this here, and also the steps involved to implement this change. I suppose some CI/CD setups will have to change at the minimum.

[mbenz89]

We could move it to "secure-software-engineering" or some new organization that we establish for that purpose, e.g. a separate "soot" organization. This might make sense, actually, so that we can also host Soot extensions there if we want.

If we go with "secure-software-engineering", we might eventually run into the same problem, i.e., we will not have the possibility to manage fine-grained access control. It might make sense to have a clear separation here.

Let's discuss this here, and also the steps involved to implement this change. I suppose some CI/CD setups will have to change at the minimum.

What currently comes to my mind:

• Soot-ci user will have to be an owner of the new organization

• A new pipeline job has to be created in Jenkins (other ci config stays as is + Build Job configuration is located in the repository itself by now, i.e., in the Jenkinsfile)

• All hard-coded urls in the Wiki, pointing to the old domain, need to be updated.

Regarding the wiki urls, I do not know how many are affected. I guess this would be the biggest task of the three.

Are you planning to also change the package naming of soot? This would break compatibility with existing clients. Additionally, we would also have to register the new group id at Sonatype got to get access rights for pushing to Maven Central. I'm not against it per se, though.

I'll think a bit more if I missed some things...

[ericbodden]

If we go with "secure-software-engineering", we might eventually run into the same problem, i.e., we will not have the possibility to manage fine-grained access control. It might make sense to have a clear separation here.

I agree. So let's create a Soot organization then.

• Soot-ci user will have to be an owner of the new organization

That should be easy.

• A new pipeline job has to be created in Jenkins (other ci config stays as is + Build Job configuration is located in the repository itself by now, i.e., in the Jenkinsfile)

One question in this context: might it make sense to migrate the jenkins setup to Github actions and build things there? From what I understand, this would simplify access control to the build.

• All hard-coded urls in the Wiki, pointing to the old domain, need to be updated.

Regarding the wiki urls, I do not know how many are affected. I guess this would be the biggest task of the three.

I guess this can be handled by editing the Wiki files offline using sed or so.

Are you planning to also change the package naming of soot? This would break compatibility with existing clients. Additionally, we would also have to register the new group id at Sonatype got to get access rights for pushing to Maven Central. I'm not against it per se, though.

We should probably do that for the next major release.

[mbenz89]

One question in this context: might it make sense to migrate the jenkins setup to Github actions and build things there? From what I understand, this would simplify access control to the build.

That indeed might be a good idea. We would not rely on our own infrastructure anymore since ci is directly in GitHub and artifacts are deployed to maven central. The drawback here is that we will have to invest a bit more time to get everything working with the GitHub actions system.

I guess this can be handled by editing the Wiki files offline using sed or so.

Good point.

We should probably do that for the next major release.

What's your preferred package/group id? They do not necessarily have to match but it's advisable to be consistent here.
We will have to testify to Sonatype that we own the group id's domain or go with something more generic like io.github.soot.

[ericbodden]

That indeed might be a good idea. We would not rely on our own infrastructure anymore since ci is directly in GitHub and artifacts are deployed to maven central. The drawback here is that we will have to invest a bit more time to get everything working with the GitHub actions system.

I think we should do it, given that we'd need to update things anyway.

I guess this can be handled by editing the Wiki files offline using sed or so.

Good point.

I can take care of this if you like. I would patch this in a separate branch for now.

We should probably do that for the next major release.

What's your preferred package/group id? They do not necessarily have to match but it's advisable to be consistent here.
We will have to testify to Sonatype that we own the group id's domain or go with something more generic like io.github.soot.

We could also try to get a new, separate domain such as soot.org. I can look into this.

[ericbodden]

Looping in @StevenArzt

[mbenz89]

I think we should do it, given that we'd need to update things anyway.

Agreed. I'll do that as soon as I find a bit of time for it. Maybe we should do this before moving the project so that everything works without disruptions (assuming the actions will be moved with the repository).

I can take care of this if you like. I would patch this in a separate branch for now.

Nice, thanks!

We could also try to get a new, separate domain such as soot.org. I can look into this.

That would be great!

[StevenArzt]

I agree that we should have a distinct organization for Soot on Github. Linking a large open-source project to an individual organization always leads to hassle when people move between organizations, or when organization add new stakeholders, that suddely end up having access to repositories for which they shouldn't.

I would also recommend registering a proper domain for Soot, such as soot.org. Having an own domain does not only help with claiming ownership of a group ID in the Maven central repository, but also gives us a place from which to organize things. If we only have a Github.io web page for now, we can just point our DNS record from the new domain there. If we later want to have a proper website, all we need to do is change the DNS record.

[ericbodden]

It seems we are in agreement - great! The github organization "soot" is already taken, so I have registered "soot-oss" instead. I will invite you as owners and then we can add other later once the project has been set up.

[ericbodden]

Regarding domains I saw that most soot.* domains are already taken, but there is now also a TLD named "oss", so maybe soot.oss would actually be sensible?

[ericbodden]

I have updated the Wiki pages, see: #1310
My suggestions for next steps:

1. add to the soot-oss organizations those other people that need access. @mbenz89 @StevenArzt can you help with that?
2. Announce the move on the Soot mailing list, wait for at least a week. I can do the posting.
3. Register new domain. I can do that.
4. Move build infrastructure to Github actions. Any volunteers?
   • Update pushing of artifacts to MavenCentral to use new domain as package name
5. Move the project, patch the Wiki, potentially also webpage see Update Soot Wiki to new organization #1310

Does that sound good to you?

[ericbodden]

Ping

[StevenArzt]

The plan sounds good. I'm currently busy with patching FlowDroid, though.

[mbenz89]

add to the soot-oss organizations those other people that need access. @mbenz89 @StevenArzt can you help with that?

Sure.

Move build infrastructure to Github actions. Any volunteers?
Update pushing of artifacts to MavenCentral to use new domain as package name

I could do that. Since I am currently a bit short on time due to a rollout for a customer, this might have to wait a few days. Otherwise, maybe someone from the research group has interest in contributing to Soot.

Btw. we also need to change the place where we publish the javadocs and options if we move away from our own infrastructure. We could probably publish those inside the repo itself after each build and additionally publish it as package with each release.

Does that sound good to you?

Sounds great!

[StevenArzt]

Btw. we also need to change the place where we publish the javadocs and options if we move away from our own infrastructure. We could probably publish those inside the repo itself after each build and additionally publish it as package with each release.

Does that sound good to you?

Sounds great!

We need to be careful to not bloat the repository. If we end up with many builds (very likely), we might have many automatic commits that increase the size of the repository. Can't we handle the Javadoc like an artifact and host it on some Nexus / whatever repository?

[mbenz89]

I see your point, I also thought about that.
Since we do not want to use our infrastructure for hosting the builds anymore, I would prefer to also don't do that for documentation.

A jar containing the javadoc will be published to Maven central. However, neither the docs nor the options will be directly browsable this way.
We could probably use another branch/repo or the github.io site for this. I'll have a look how other projects handle this, I'm sure there should be a convenient way.

[StevenArzt]

I can see your point if you don't want to host anything anymore. The large OSS projects have some central hosting facility, though - think of Eclipse or Apache. For smaller projects, I'm not sure how they handle that.

[patricklam]

I've been looking at a bunch of projects for some research that I've been doing. Many projects these days seem to tag releases on GitHub and call it a day. (You can then download the tagged releases automatically).

…

On Thu, Mar 12, 2020, 6:49 AM Steven Arzt ***@***.***> wrote: I can see your point if you don't want to host anything anymore. The large OSS projects have some central hosting facility, though - think of Eclipse or Apache. For smaller projects, I'm not sure how they handle that. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOKE5TCSPIUA32E6QGS22DRG7FJHANCNFSM4K72VUPA> .

[ericbodden]

I've been looking at a bunch of projects for some research that I've been doing. Many projects these days seem to tag releases on GitHub and call it a day. (You can then download the tagged releases automatically).

@patricklam I am not sure I understand. How would that help with getting a browsable documentation? From what I understood the concern is that the generated docs are large and we do not want to store them in the repo for every build.

We could push the docs to some website. If we register a domain for Soot then we can just as well push the content there. gh-pages seems unsuitable to me because it itself is versioned and thus would still cause a blowup. After all it's just another branch in the same repo.

[ericbodden]

I could do that. Since I am currently a bit short on time due to a rollout for a customer, this might have to wait a few days. Otherwise, maybe someone from the research group has interest in contributing to Soot.

Thanks Manuel. Some jobs certainly could and should be taken on by others but I am not sure the build system is the right job for people who are not too familiar with the same already. I would not trust too many people with this :-)

[mbenz89]

We could push the docs to some website. If we register a domain for Soot then we can just as well push the content there.

We can do this or go with our current solution of self-hosting and maybe point the new domain to our servers depending on if we get some webspace with the domain.

Thanks Manuel. Some jobs certainly could and should be taken on by others but I am not sure the build system is the right job for people who are not too familiar with the same already. I would not trust too many people with this :-)

Well, thank you, I guess. :)

[ericbodden]

Update from my side: I have asked our admin to see if we can reserve the domain soot-oss.org. The domain soot.oss would be nicer but has already been bought by domain grabbers, as have all other soot.* domains. I really fail to see why this is still legal but apparently it is...

[mbenz89]

I've set up an initial GitHub actions file which runs build, test, style- and license checks: https://github.com/Sable/soot/blob/develop/.github/workflows/ci.yml

The deploy part is still missing but shouldn't be too much hassle. I got to say, that I start liking the GitHub actions dsl. It's easier to write and read than the Jenkins one...

On a side note, we will have to adapt Heros, AXML, and Jasmin (if we want to keep it) as well. We will be able to copy-paste most of the build file, however.

[ericbodden]

Sounds great @mbenz89. How do you suggest we proceed then?

[mbenz89]

I planned some time to implement that last part tomorrow evening and copy it over to the other repositories. We will still use our build servers for research group project and for hosting the options/jdoc of Soot. We will neither host nor build official Soot components on that infrastructure anymore. When the actions pipeline is ready, we can move Soot to the new organization and apply your wiki changes and everything should still work.

Users who cloned or forked Soot outside of Github will have problems with an unreachable remote. We cannot do much against that, however. I think we should be fine otherwise.

[mbenz89]

Update from my side: I implemented the deploy step for GitHub actions. I'm still having some problems deploying the JavaDoc and Options to our server due to authentication issues with the Jenkins user on our build machine. I'll contact our admin to help with that. Until then I'll disable the deplyoment of the documentation.

I've already disabled the use of our own infrastructure and updated the READMEs accordingly. If we want to move Soot before having a working deployment for that documentation, we are good to go. I still got to implement actions for the other repositories before we move them, though.

[ericbodden]

Thanks @mbenz89! Did you get anywhere with the JavaDoc and Options?

Also, which "other repositories" do you mean? heros and jasmin? I think it would make most sense to move them all together...

[ericbodden]

On another note, I have now reserved soot-oss.org. @mbenz89 what do we have to do to prove ownership for the maven ID?

[mbenz89]

Thanks @mbenz89! Did you get anywhere with the JavaDoc and Options?

Due to my absence during the last week, I've not made progress here. Our admin is notified, however, and I expect this to be resolved in the next days

Also, which "other repositories" do you mean? heros and jasmin? I think it would make most sense to move them all together...

Exactly. Inclusive our axml fork.

On another note, I have now reserved soot-oss.org. @mbenz89 what do we have to do to prove ownership for the maven ID?

We will have to make a ticket in their issue tracker requesting an ID. I can do that since I already did this for our current ID, or you can do it if you like to. We will have to testify that we own the ID afterward by sending a mail from the domain or setting a specific DNS entry. The whole procedure usually just takes 2-3 days.

Our new group id would then be org.soot-oss, right?

[ericbodden]

Our new group id would then be org.soot-oss, right?

Correct, thanks!

[ericbodden]

@mbenz89 any progress on this? BTW should we send an announcement to the Soot list to give people some heads-up?

[mbenz89]

@ericbodden There is some progress. Our Admin has just created a new server to which a special user should have upload rights. For the Documentation deployment, I have to set up a webserver there that serves the documentation and modify the upload procedure a bit to get this working.

Missing:

• Documentation deployment to new server + URL change in wiki and Readme.md

• Implement actions for Heros, Jasmin and AXML

I'll probably not be able to do this before next Weekend.

BTW should we send an announcement to the Soot list to give people some heads-up?

That shouldn't hurt I guess :).

[mbenz89]

To testify that we own the domain, we have to do one of the following:

Do you own the domain soot-oss.org? If so, please verify ownership via one of the following methods:
· Add a TXT record to your DNS referencing this JIRA ticket: OSSRH-56893 (Fastest)
· Setup a redirect to your Github page (if it does not already exist)

@ericbodden Could you do that?

[ericbodden]

To testify that we own the domain, we have to do one of the following:

Do you own the domain soot-oss.org? If so, please verify ownership via one of the following methods:
· Add a TXT record to your DNS referencing this JIRA ticket: OSSRH-56893 (Fastest)
· Setup a redirect to your Github page (if it does not already exist)

@ericbodden Could you do that?

Done! (through DNS record)

[ericbodden]

I'll probably not be able to do this before next Weekend.

BTW should we send an announcement to the Soot list to give people some heads-up?

That shouldn't hurt I guess :).

Okay I will do this right now.

[mbenz89]

Update:

• I fixed quite some problems with the documentation problem in the GItHub actions file (095f675)
• I'm still in contact with our admin for proper access to the build server for deploying the documentation
• I will port the other repositories as soon as Soot is working 100% with GitHub actions to save me from fixing some configuration issues with the documentation deploy in all repositories afterward (This should hopefully be a small task when we have one properly working config)
• I acknowledged to OSSRH that we have set a DNS record. We should hopefully get ownership of the new group-id soon

[ericbodden]

Since the DNS did not work, I now set up a redirect to: https://github.com/Sable/soot

[mbenz89]

Thanks, Eric! Could you please change the redirection to https://github.com/soot-oss?
Since we want the soot-oss.org domain, we have to testify its ownership.

[ericbodden]

@mbenz89 the group ID was now approved. Are we still missing anything else to conduct the change?

[mbenz89]

No, I don't think so. I just did not find the time to conduct the missing changes this weekend. I might give it another try next weekend.

We still need to:

• Setup doc/options deploy to new swt-build server

• Setup a server application that serves the doc/options

• Duplicate a working setup for heros/jasmin/axml

I guess we change the group id of soot as soon as we move the repo to the new organization. Should I adapt the package names as well for consistency reasons? Keeping the packages as is would not require any client changes. It might be a bit misleading still...

[ericbodden]

Hi @mbenz89. I don't want to bother you but nonetheless I was wondering whether you were able to make some progress on this.

[mbenz89]

Hi @ericbodden. Unfortunately only very little. I'm still having problems accessing the new build server swt-build to which we want to deploy the documentation. I'm in contact with our admin regarding the issue. I'll give it another try when I got proper access.

[ericbodden]

@mbenz89 on June 15th I will be giving a SOAP-talk about Soot. It would be great if I could communicate the updated URLs by then.

[mbenz89]

@ericbodden Sammy and I were able to fix the access yesterday. I'll give it another try this weekend. I guess the 15th should be doable!

[mbenz89]

Update: I tried again today and it seems that the connections errors were only fixed temporarily. I need proper access to the new deployment server before I can proceed :(.

[ericbodden]

Update: I tried again today and it seems that the connections errors were only fixed temporarily. I need proper access to the new deployment server before I can proceed :(.

What does proper access mean? Is there anything I can do to help?

[mbenz89]

@ericbodden I just moved the repository. I'll now adapt the group id and the deployment process to work with it.
You could apply your changes to the wiki now.

[mbenz89]

We also need to change the redirection of soot-oss.org to the github pages page...

[mbenz89]

Update:

• I updated all the links and fixed some other left-over issues in develop, master and gh-pages branches.

• Update to wiki is still missing

Unfortunately, I cannot publish the artifact to Maven Central currently since my user does not have the correct rights set for org.soot-oss. I've contacted the support for this. Usually, they are very fast, so I expect this to be solved sometime on Monday. I guess we should wait with an update to the mailing list until the artifact got properly deployed. However, I also think its totally fine to announce the repository movement at SOAP on Monday.
For now, the maven coordinates in the README are thus still documented as ca.mcgill.sable.soot so that the artifact is obtainable. I'll change same as soon as the deployment is working.

I'll successively move the other repositories and migrate them to GitHub actions, as soon as the option/jdoc deployment is working properly for Soot

[ericbodden]

We also need to change the redirection of soot-oss.org to the github pages page...

Done!

[ericbodden]

@mbenz89 thanks a lot for this! Would it now be save to move jasmin and heros, too?

[mbenz89]

We could. I would rather like to wait with it until we have a proper deployment of the Options and Documentation and do a batch migration then. Would that work for you?

Have you updated the links in the wiki?

Btw. The artifact deployment to Maven Central now works properly and I've adapted the Maven coordinates in the READMEs accordingly.