svn viewer fully integrated with the website

[mateuszviste]

svn is the heart of all SvarDOS developments, documentation and distribution of packages.

Currently I have set up a "websvn" instance on a virtual host, so it is possible to look up svn commits etc via a web browser. websvn is not super intuitive to me, and its look is also very different from the SvarDOS website.

The idea here would be to have a simple svn viewer integrated right in the website. It would share the same styling than the rest of the website, and would be kept in svn along with the rest of the files so it would require no configuration.

Such interface would need to provide:

• a view of all the files and directories in svn (with a configurable revision number)
• text viewer for text (and source code) files, with an optional "blame" view
• source code coloring would be nice if easy to add
• a view of all svn commits, where each commit would show the author, timestamp and log of the commit
• clicking on a commit shows the list of modified files, and for modified files another link allows to see a diff

The PHP code would rely either on svnlook or use the PHP SVN bindings. The latter would be much more comfortable, but they are not part of the PHP install on Debian... requires some "PEAR" magic (and I'm not even sure these bindings are still available for PHP 8 anyway).

[roytam1]

what about syncing it to github?

[mateuszviste]

Hi Roy, not a bad suggestion, but I really prefer to stick with svn. :)

[roytam1]

for now I do it myself for my own use: https://github.com/roytam1/SvarDOS

[mateuszviste]

for now I do it myself for my own use: https://github.com/roytam1/SvarDOS

Hi Roy, would you mind explaining how you do it? I mean, the exact git syntax and what not? Independently of this issue perhaps I could set up some automated read-only mirror here on github. I figure it could help SvarDOS get some extra exposure.

[roytam1]

Hi Roy, would you mind explaining how you do it? I mean, the exact git syntax and what not? Independently of this issue perhaps I could set up some automated read-only mirror here on github. I figure it could help SvarDOS get some extra exposure.

https://gist.github.com/rickyah/7bc2de953ce42ba07116#cloning-the-svn-repository
https://gist.github.com/rickyah/7bc2de953ce42ba07116#getting-the-latest-changes-from-svn

[mateuszviste]

Apparently the OSDN service finally died a couple of days ago. Until now the OSDN subversion server was being used to keep a backup of the SvarDOS repository. Now that OSDN ceased to respond, we no longer have a backup of SvarDOS svn. Time to look for some new backup location.

[boeckmann]

How is this backup of the SVN performed? Via svnadmin dump / load? I could provide a container or VM for this, and give you access to it...

[mateuszviste]

How is this backup of the SVN performed? Via svnadmin dump / load?

No, it's an svnsync action I execute manually on my laptop using this script:
http://svn.svardos.org/filedetails.php?repname=SvarDOS&path=%2Fsync-mirrors.sh

I could provide a container or VM for this, and give you access to it...

That would be awesome. The repo mirror needs about 700M of disk space currently (without packages, these are mirrored on helix already). I have described the exact procedure of setting up a SvarDOS mirror here (scroll to the bottom):
http://svn.svardos.org/filedetails.php?repname=SvarDOS&path=%2Fdoc%2Fbuild%20server.txt

[mateuszviste]

It takes 700M because its history contains some binary packages from the time when packages and code were in a single svn repo. Now packages have their own repo, but what has been done has been done. There is surely a way to curate mirrors from these historical blobs, but I did not investigate it.

[boeckmann]

@mateuszviste can you mail me your public ssh key so that I can give you access to the VM?

[boeckmann]

I have setup svnsync, which runs every 15 minutes as a cron job on svn.boeckmann.io. The repository is not yet publicly exposed, but @mateuszviste you may SSH login via the svnmirror user to get access to the repo in case of an "emergency". The repo lives under /home/svnmirror/repos/svardos. I added your SSH key, so you should be able to login. I can make the repo read-only visible to the world, if it is needed.

[mateuszviste]

I confirm it all works - I have ssh access, the cron seems good and the on-disk repo is indeed up to date. Awesome, thanks!
No need to make this publicly open I think, it's just a mirror after all.

Just wondering, though - maybe we could use this cool svn mirror to replicate the source code to git, like what Roy was suggesting a couple of months ago? This would make it easier to spot if one day the mirror stops being up to date for whatever reason, and it would also act as an additional backup. Plus it might give SvarDOS some extra exposure.

[mateuszviste]

if you do make this repo public some day let me know - I will then mention it on the SvarDOS website

[boeckmann]

Just wondering, though - maybe we could use this cool svn mirror to replicate the source code to git, like what Roy was suggesting a couple of months ago? This would make it easier to spot if one day the mirror stops being up to date for whatever reason, and it would also act as an additional backup. Plus it might give SvarDOS some extra exposure.

Sure, I will try to get local SVN->Git mirroring on the VM working in the afternoon. When this works we can create a Git repo here that receives the changes. Should not be much work to implement.

[boeckmann]

There are three authors in the SvarDOS SVN log I am not sure I can include their mail address in the git repo conversion. In contrast to SVN, Git wants User Name <email> scheme for committers. @cardpuncher and @bttrx may answer here if a mail address may be used or if it should be substituted by an anonymous alias. @mateuszviste if you may ask uzemario.dantas, as I have no contact information.

[mateuszviste]

Maybe best would be to put some generic nonsense for all commits without distinction so they all appear being from SvarDOS developers <not_a_mail@svardos.org> ? This way if in the future someone else appears in the svn log there will be no issue with mapping it again.

[boeckmann]

I now have also mirrored the svardos-pkg SVN repository.

I made a test conversion of the SvarDOS SVN repo to Git. This worked, and upload to a (yet private) Github repo went smooth. However, git svn does not include the externel references to the svardos-pkgs repo, resulting in broken links for the packages-core. Not yet sure how to handle this. I will perform some investigation on how this can be solved...

[cardpuncher]

On 26.09.2024 16:20, Bernd Böckmann wrote: . @cardpuncher <https://github.com/cardpuncher> and @bttrx <https://github.com/bttrx> may answer here if a mail address may be used or if it should be substituted by an anonymous alias.

Hi, I would prefer an anonymous alias to avoid receiving spam on my e-mail address.

[boeckmann]

Hi, I would prefer an anonymous alias to avoid receiving spam on my e-mail address.

I changed the script to only include the SVN user names. Email is set to an empty address.

[boeckmann]

@mateuszviste can we make two new repositories here, for the SVN svardos and svardos-pkgs repository mirrors?

[boeckmann]

At best with "mirror" in the names, like svardos-mirror and svardos-pkgs-mirror

[mateuszviste]

is it really useful to expose svardos-pkgs through github? it's a bunch of binary files... plus it might grow with time, not sure github will be happy with such use. Esp. since many of these files are closed-source blobs.

for the svardos "system" repo I was thinking about renaming our current "bugz" repo to "core" - does that make sense?

[boeckmann]

We can leave out the pkgs. But the main repo is also not that small, because packages dir once contained the real files. I could exclude the packages dir from the conversion alltogether. This would make the repo much smaller, but would somewhat break the older commits. But if this is Github mirror should mainly be for documentation I think we could do this. We can call the repo "core".

[mateuszviste]

We can leave out the pkgs. But the main repo is also not that small, because packages dir once contained the real files. I could exclude the packages dir from the conversion alltogether. This would make the repo much smaller

I think it is a good idea. At some point I will look at how to strip this ancient history from the svn repo, but for the time being just cutting it out when mirroring to github would be perfect.

We can call the repo "core".

I will look into that now.

[boeckmann]

Only one thing to consider that comes in my mind right now: Not quite sure it would be wise to push the svardos SVN into the same Git repo as the bugs. If we at any time have to wipe the git repo and set it up again because we noticed some flaws in the mirroring, chances are we would also loose the bug issues, as I am not sure that there is a way to wipe the git repo completely without deleting the github repo (issues inclusive). At least we should investigate this before pushing the files...

[mateuszviste]

I am not familiar with how git works, but apparently it is possible to overwrite a repo by "force pushing" new content:

# set up a remote
git remote add origin <url-of-remote>
git branch --set-upstream master origin/master

# push new history
git push -f

If that's correct (is it?), then no need to fiddle with the github settings of the repo (and risking to loose issues).

[roytam1]

If that's correct (is it?), then no need to fiddle with the github settings of the repo (and risking to loose issues).

yeah, but old content may still able to be fetched, unless telling github to flush the cache:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository#fully-removing-the-data-from-github

[boeckmann]

Would be good if at least the symbolic-link versions of the .svp packages under packages-core stay intact, and otherwise remove the non-link .svp files. I think this can be achieved with the git-filter-repo command. Will check this out...

Regarding the wipe of the repo, I am not sure that simply force pushing a new history to Github will clean all the data (I think Git calls this garbage collection). Seems that locally garbage collection can be initiated manually. But at the Github repo, I do not know yet...

[boeckmann]

I made a test repo at https://github.com/boeckmann/svardos-core/
I nothing bad is found with it, I can use the same script to make this available uner the SvarDOS oranization.

[boeckmann]

The script used:

#!/bin/sh

export PATH=/home/svnmirror/bin:$PATH

# mirror SVN repositories
svnsync sync file:///home/svnmirror/repos/svardos >/dev/null || echo 'failed to sync svardos SVN'
svnsync sync file:///home/svnmirror/repos/svardos-pkgs >/dev/null || echo 'failed to sync svardos-pkgs SVN'

# mirror to Git
#git svn clone svn://svn.svardos.org/svardos --authors-prog=svn-to-git-author.sh
cd /home/svnmirror/git-repos/svardos ; git svn fetch --authors-prog=svn-to-git-author.sh >/dev/null && git reset --soft refs/remotes/git-svn >/dev/null || echo 'failed to sync svardos Git'

cd /home/svnmirror/git-repos/svardos ; git push --mirror origin >/dev/null 2>&1 || echo 'failed to push svardos Git'

[boeckmann]

Script to filter out the .svp files:

#!/bin/sh
git filter-repo --force --commit-callback '
commit.file_changes = [
    change for change in commit.file_changes
    if not ((change.mode == b"100755" or change.mode == b"100644") and change.filename.endswith(b".svp"))
]
'

[mateuszviste]

looks perfect to me :) very cool!

[boeckmann]

So how to proceed with this? Git synchronization seems to work, but like I said, I would be more comfortable to push this into its own repo separate from the issues. Having slept a few days nights over it, it turned out I actually like the old bugz repo name for the kinds of issues it deals with more than core. For "outsiders" I think it is more clear that "bugz" is a central contact point of reporting all kind of issues than "core". But its mainly @mateuszviste project, so I will push to where ever he wants it to be pushed :)

[boeckmann]

Also, there are already links to the bug tracker out in the wild. Renaming the repo broke all of them.

[boeckmann]

Also, there are already links to the bug tracker out in the wild. Renaming the repo broke all of them.

Seems that Github makes a redirect from the old name, so thankfully the old links are still valid... Did not expect this...

[mateuszviste]

I would be more comfortable to push this into its own repo separate from the issues.

I also find the concept of a "throwable" repo attractive.

Having slept a few days nights over it, it turned out I actually like the old bugz repo name for the kinds of issues it deals with more than core.

Same here. I keep typing "bugz" in my browser address bar... so we are back with bugz now.

But its mainly @mateuszviste project, so I will push to where ever he wants it to be pushed :)

I talked with him under the shower and he totally agrees with you. Bugz = issues. Core = repo mirror.

[boeckmann]

I talked with him under the shower and he totally agrees with you. Bugz = issues. Core = repo mirror.

Well then so shall it be! Did he also consider giving me the rights to add the needed deployment keys? Or adding this as deploy key with write rights?

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILzOcBVCFAFFtj8clmRAB7mhjF8Z0J3DJtFjgx88dLsN svnmirror@bernd-vm

[mateuszviste]

Github does not like this ssh key, says "key already in use" (whatever that means). But you are the admin of the repo now, so hopefully you will figure it out :)

[mateuszviste]

Seems that the github mirror works, and updates automatically. It is awesome, thank you Bernd!
I will add a link to it on the svardos website tomorrow and then close this issue.
I wonder if it makes sense to keep the web frontend at svn.border6.com alive. Seems to attract mostly scanners and some annoying spider bots.

[boeckmann]

I will add a link to it on the svardos website tomorrow and then close this issue.

Great! Can you also add a link to the kernel repository?

I wonder if it makes sense to keep the web frontend at svn.border6.com alive. Seems to attract mostly scanners and some annoying spider bots.

Not sure what svn.border6.com is. Never visited that site. I just tried to open it but got a timeout. I assume it has nothing todo with svn.svardos.org?

[mateuszviste]

Great! Can you also add a link to the kernel repository?

Done

Not sure what svn.border6.com is. Never visited that site. I just tried to open it but got a timeout.

My brain glitches sometimes with inter-thread memory leaks. It's tightly sealed and out of warranty now, sadly not much I can do.

I meant svn.svardos.org. Currently it uses websvn and I really do not like it. It's a relatively big piece of code that I do not trust much, plus it is constantly hammered with some weird scripts from around the world, occasionally causing performance issues on the server. github provides a much more intuitive interface, so I am not sure there is any added value in exposing svn.svardos.org over an ugly http front.

I'd still like to have svn somehow integrated with the main svardos.org website, but now that we have this nifty github export it does not need to be super functional. Maybe just a list of commits with a single button to see a diff. Could also be exported via RSS so one can be notified whenever a commit is made (that's the only websvn feature I actually use).

[boeckmann]

I actually like the WebSVN interface, but I you think that this imposes a security risk then maybe it is better to deactivate it.

[boeckmann]

SvarDOS uses a fork of the EDR kernel, whose development is kept on the EDR github repository.

Maybe you replace the first "EDR" by "Enhanced DR-DOS"? I think many people do not know what "EDR" stands for.

[mateuszviste]

Maybe you replace the first "EDR" by "Enhanced DR-DOS"?

Done.

I actually like the WebSVN interface

I did not know you use it. Let's keep the status quo for the time being then.

[roytam1]

I you think that this imposes a security risk

WebSVN can bring you denial of service as people can tell WebSVN to run some svn commands that can take lots of CPU time.

[mateuszviste]

I you think that this imposes a security risk

WebSVN can bring you denial of service as people can tell WebSVN to run some svn commands that can take lots of CPU time.

That's indeed what happened to me a few weeks ago. svn.svardos.org was being intensively hammered (like thousands of requests per minute) by facebook. CPU load avg spiked to 60+, ssh was barely responsive. At first I thought I'm under attack, but apparently it's just some human incompetence. I'v blacklisted the facebook bot through this mod_rewrite rule and the problem did not reappear:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^facebookexternal.* [NC]
RewriteRule ^ - [F,L]

Then earlier this year we had another issue (#57) with websvn - it was leaking temporary files and filling disk space.

So yes, these are the reasons I'm starting to be a little bit weary of websvn :P

[boeckmann]

I did not know you use it. Let's keep the status quo for the time being then.

Yes I used it as this was the easiest way for me to look at SVN commit diffs etc. But now that this is mirrored on Github I may equally well use that, or actually clone the Git repo and use my local tools on it (mainly Sublime Merge). So while I actually like the WebSVN interface, I think I would not miss it much in case you abandon it...