Drumkit.xml files should be upgraded at build-time #1481
Comments
|
P.S. I'm not interested in Ubuntu myself, but I want to do what I can to provide downstream users with a great experience. You know, because these users tend to be stuck on the same Hydrogen version for two years or more ;-) If HEAD is in a good state for a point release, then maybe that's an option too. No worries or pressure of course, it's just that time of year on an even-numbered year! |
|
Nice timing! I just skimmed through a user's log the other day and was quite confused to find those kits being installed on system level and not in the user's HOME.
Hydrogen itself does feature a dialog for downloading (the most recent version of) all drumkits. And it does so since the beginning of our git history in 2007. There are two kits that should be bundled with the data folder installed on system-level in order for the user to get started right away and we do our best to keep them up-to-date. On Devuan Beowulf - which most probably uses the Debian repo as upstream - I do see two packages bundling various kits: From my point of view dropping these two bundle packages would be best. Else we have to deal with kits being present at multiple locations, can not upgrade kits as they are read-only, and do not have control over the versions shipped. |
|
Hi, @theGreatWhiteShark, you seem to be correct there. The |
|
theGreatWhiteShark ***@***.***> writes:
Nice timing! I just skimmed through a user's log the other day and was quite confused to find those kits being installed on system level and not in the user's HOME.
> All drumkit.xml files should be "updated" when the package is built, and this seems like something that should be done by default in the upstream rather than worked around during build time in every distribution. Alternatively maybe these files shouldn't be installed at all, but should instead be generated in $HOME.
Hydrogen itself does feature a dialog for downloading (the most recent version of) all drumkits. And it does so since the beginning of our git history in 2007. There are two kits that should be bundled with the [data](https://github.com/hydrogen-music/hydrogen/tree/master/data/drumkits) folder installed on system-level in order for the user to get started right away and we do our best to keep them up-to-date.
On Devuan Beowulf - which most probably uses the Debian repo as upstream - I do see two packages bundling various kits: `hydrogen-drumkits` and `hydrogen-drumkits-effects`. Do you know why or since when they exist? Maybe this is something predating Hydrogen's internal download dialog. Don't know.
Sebastian Moors ***@***.***> writes:
Hi,
@theGreatWhiteShark, you seem to be correct there. The `hydrogen-drumkits` packages seems to date from [2006](https://metadata.ftp-master.debian.org/changelogs//main/h/hydrogen-drumkits/hydrogen-drumkits_2015.09.28-1_changelog) (Hydrogen 0.9.3), the sound library manager got introduced in 0.9.4. But i'm not sure if the online import was there in a different form before..
One has to consider that back then not everyone had fast internet access and it was nice when such packagescould be found on one of the cdroms of your favourite distribution :)
Yes :-) In Debian we also (still) accommodate users with metered
internet connections. We have many users to drive into town to download
packages, then back up the mountain, so to speak, to upgrade their
systems. Unfortunately it's impossible to say how many of this group
are Hydrogen users.
From my point of view dropping these two bundle packages would be best. Else we have to deal with kits being present at multiple locations, can not upgrade kits as they are read-only, and do not have control over the versions shipped.
Yes, read-only packager-level control is the expectation on Debian
systems, and I concede that this is different from how things work on
most (all?) other platforms. Were additional DFSG-free drumkits
required to fulfill expectations for real work, those could be provided
via the backports suite/repo.
Unfortunately there's not enough time to properly (11 days) test your
proposal, as it represents a major shift in the package. The Debian
package is also patched to use a DFSG-free drumkit source via
`2001_avoid_live_non-free_drumkits.patch`. So,
1. Users who expected drumkits to be available using installation media
on metered internet would find that all their drumkits have disappeared.
2. I won't have time to adequately test freepats.zenvoid.org's drumkit
source to see if it would be suitable as the sole provider of drumkits.
3. There's also the issue of provider bandwidth costs for an unbounded
number of users. Currently the Debian mirror network, and the new
deb.debian.org CDN provides these, and the package is ~230MB/user. I'm
guessing there are at least 3x Ubuntu (and derivative) users for every
popcon-registered Debian user, and if they all downloaded all the
replacement drumkits online (eg: if you proposal was implemented) that
could be 975 200MB in one month, which I doubt zenvoid.org is willing to
pay for and I suspect hydrogen-music.org wouldn't be happy with either
(eg: if 2001_avoid_live_non-free_drumkits.patch was dropped).
4. Concerning dropping that patch, that's a big change from a Debian
perspective (Debian is a legally and ideologically free/libre
distribution) and would require what is likely to be a long discussion,
and it could conceivably require moving Hydrogen from main
(DFSG-compliant Debian-proper) to contrib (FLOSS software that depends
on non-free assets like game graphics, sound effects, fonts, etc). Were
that the case 11 days is not enough time.
This is why I think dropping the hydrogen-drumkits package is too big of
a change to push with less than 11 days of discussion and testing. "Do
nothing" is of course also a totally viable option.
|
|
Hi, first of all, let me say something about the drumkits in your package: Those drumkits we're sent to us by users who ask us to host them at our drumkit server. Most of the time, we do not touch them or develop them further. They are also not part of any kind of git repository, they live at a file storage at Sourceforge. In the past we only touched them if they had major errors. In contrast to this, you want to make shure that the xml version of the kits in your package match the version of your current Hydrogen binary. This is a different requirement/use case then we have for our hosted kits. In my view this means that the kits for your package need active maintenance and must be upgraded and tested for each new version of Hydrogen. I'm currently not active in Hydrogen development so i can't really judge if this is feasible for Hydrogen or not, but this creates some additional work for the Hydrogen team. There could also be a problem of putting also those kits in git repos at Github because of file size limitations (i have to check on that). |
|
Hi Sebastian,
Sebastian Moors ***@***.***> writes:
Hi,
first of all, let me say something about the drumkits in your package: Those drumkits we're sent to us by users who ask us to host them at our drumkit server. Most of the time, we do not touch them or develop them further. They are also not part of any kind of git repository, they live at a file storage at Sourceforge. In the past we only touched them if they had major errors.
In most cases, we also did not upgrade them to newer drumkit versions - mostly because our online drumkit service is not able to provide multiple versions of a file, which leads to compatibility problems (all versions of Hydrogen use the same drumkit server). I hope this makes it a bit better understandable why we don't upgrade all of those drumkits on our servers.
Yes, that makes sense.
In contrast to this, you want to make shure that the xml version of the kits in your package match the version of your current Hydrogen binary. This is a different requirement/use case then we have for our hosted kits. In my view this means that the kits for your package need active maintenance and must be upgraded and tested for each new version of Hydrogen. I'm currently not active in Hydrogen development so i can't really judge if this is feasible for Hydrogen or not, but this create some additional work for the Hydrogen team. There could also be a problem of putting also those kits in git repos at Github because of file size limitations (i have to check on that).
I agree, and yes, when I noticed this error (I recently adopted the
package and want to fix all its deficiencies), it seemed like a case of
bitrotting that ought to be investigated. As far as I can tell, the xml
drumkits in the Debian package have never been refreshed. To be clear,
I'm not asking that the Hydrogen team upgrade its xmls drumkits, nor
host them somewhere new--only for mechanism to non-interactively refresh
these xml drumkits. It's also not my place to assert that buildtime >
runtime for the purposes of upstream Hydrogen development.
Given that hydrogen refreshes the xml files at runtime, there must be a
function like `rebuild_refresh_drumkit_xml(drumkit_path_or_xml_path)`.
It seems like it would be enough to resolve this issue if that function
was exposed as command line argument so that hydrogen could provide this
utility function. Something like `hydrogen --refresh-drumkit`
$PATH_or_xml_PATH`. If this required running a GUI, then I'd run it
with `QT_QPA_PLATFORM=offscreen` or even xvfb-run if necessary, but it
would be really nice if it didn't require a GUI. Accepting either a
basedir of the drumkit, or the path to the xml file as an argument would
do the trick.
I'm hoping that it is trivially easy for someone familiar with Hydrogen
to expose this utility function as a command line argument.
Unfortunately the kits won't be in the expected locations, because
they'll be unpacked to a tmp build dir, which is why I'll need to
provide something like $temp_build_dir/$drumkit_name_basedir[.xml] as an
argument. That requirement/limitation should be easy to resolve too.
It would be totally unreasonable to ask for more than this with a
deadline just 10 days away ;-)
If it's not quick and easy, no worries, it can wait!
Regards,
Nicholas
P.S. I didn't know you were no longer active in Hydrogen development.
Happy retirement, and/or best wishes for whatever else you're working
on!
|
I think that's feasible and it also might be helpful while resolving other issues. I would add this to the current |
|
theGreatWhiteShark ***@***.***> writes:
> I'm hoping that it is trivially easy for someone familiar with Hydrogen
> to expose this utility function as a command line argument.
> Unfortunately the kits won't be in the expected locations, because
> they'll be unpacked to a tmp build dir, which is why I'll need to
> provide something like $temp_build_dir/$drumkit_name_basedir[.xml] as an
> argument. That requirement/limitation should be easy to resolve too.
I think that's feasible and it also might be helpful while resolving other issues.
That's wonderful news, and yes, that was/is my hope; at the very least,
it seems like it could added as a CI test for drumkit.xml-related
functionality. I'm thinking about using this as a method for notifying
when then the hydrogen-drumkits Debian package needs to be refreshed for
use with a new Hydrogen version. Yes, I feel bad that no one noticed
this for many years :$
I would add this to the current `master` branch and not v1.1.1. Is this okay with you?
Yes that totally ok! :) I'll definitely make an attempt to backport the
commit[s] and, if successful, can start a 1.1.1 (or 1.1.x) branch on my
remote, then submit it for review with nice changelog entries etc.
'hopefully the drumkit.xml work done on master in early 2021 won't make
this too tricky ;)
|
Backporting shouldn't be too difficult as there were not much changes in this part of code. But I can also provide you a patched version of v1.1.1 in a custom branch. Btw. I'm almost done. I think I can give you a working version tomorrow afternoon latest. |
|
@sten0 I created a PR in #1490. It still needs a little bit of testing (and documentation) but the API should be stable. I tested the upgrade for all the .h2drumkit files hosted at our SourceForge page, which should be a superset of what is contained in the Debian package and - at least as far as my unit test is concerned - it should work for all of them. But please have an open eye on the results. |
|
Alright. I'm done. I also rebased all commits of #1490 on tag 1.1.1 in branch drumkit-cli-1.1.1. |
|
Sorry for the delay. Life. Here's an update, working from a forgotten
draft:
theGreatWhiteShark ***@***.***> writes:
Hi @theGreatWhiteShark !
@sten0 I created a PR in #1490. It still needs a little bit of testing (and documentation) but the API should be stable.
Thank you, that's wonderful news :-)
I tested the upgrade for all the .h2drumkit files hosted at our [SourceForge page](https://sourceforge.net/projects/hydrogen/files/Sound%20Libraries/Main%20sound%20libraries/), which should be a superset of what is contained in the Debian package and - at least as far as my unit test is concerned - it should work for all of them. But please have an open eye on the results.
Yes, it's definitely a superset. At some point in the future I hope to
take a look at their licenses and see which new ones (new = post-2017
:-o) could be incorporated into the Debian package. Does SourceForge
track download counts these days? If the package ends up feeling "too
big" (probably ~325MB) then I'd want to split the collection using some
kind of rational scheme. Classic kits and kits used in the demos in the
basic collection, and then either another package for the rest, or maybe
according to some era or typical use in a genre of music. That's
orthogonal to this issue though.
Backporting shouldn't be too difficult as there were not much changes
in this part of code. But I can also
provide you a patched version of v1.1.1 in a custom branch.
Yes, I noticed conflicts that slowed me down do to lack of Hydrogen
knowledge.
Alright. I'm done. I also rebased all commits of #1490 on [tag 1.1.1]
Thanks, much appreciated! I see that I had made two mistakes when in my
private/local copy of the rebase.
More importantly though, I read some comments in eab4acc that indicated
that an absolute path to the drumkit is needed? I'm still investigating
whether this will work on Debian's automated build infrastructure. I'm
sorry I forgot to mention this potential issue...I wrongly assumed that
relative (to the pwd/cwd where hydrogen is executed) would be supported.
The drumkits collection source package will use the hydrogen package
(installed to the build system) as a utility to upgrade the drumkits,
where the path of the drumkits is relative to BUILDDIR. BUILDDIR is the
basedir of the wherever the automated builder unpacks the build, and
yes, this is randomised. I'd like to continue to investigate if it will
be possible to make this work without relative path support, and hope
that it won't be necessary. After all, the onus is on me to be clever
about stuff like this haha.
Meanwhile, for the Ubuntu LTS consideration, I realised that it would be
better to not rush an update. It seems safer to request an update once
our new solution has been tested for at least a week or two. I hope
that the Ubuntu release team will concur with a request from the Debian
maintainer especially when it has upstream support ;-)
Best,
Nicholas
|
|
Update: It seems I was suffering from cognitive bias with a dash of
ignorance. Sorry for that :$ A team member suggested investigating
"dpkg triggers" as an alternative to refreshing the drumkit.xml files at
build time. As I understand these triggers, the solution will work
thus: when a new version of Hydrogen is installed, it triggers a refresh
of the system copy of the drumkits; when a new version of
hydrogen-drumkits is installed, it of course also activates the same
function.
This method will work with absolute paths! Also, I feel good about this
solution because it's closer to standard operation of Hydrogen with
drumkits in $HOME.
What remains is Debian-specific implementation details, and I'm
currently the bottleneck since I'll need to take some time to learn how
to avoid a bunch of pitfalls to do it correctly the first time.
Thanks again for exposing this functionality, I truly appreciate it.
|
|
Hey @sten0 ,
Don't worry.
I think so. At least they are shown for the individual files when browsing through the uploaded folders. But I know frightening little about SourceForge.
I think so too. After all it's just about not flooding the console in case user decides to open Hydrogen via terminal and a minimal increase of startup time.
This is mainly because previously I did everything using OSC commands and there relative paths are not an option. But I definitely took a liking to our CLI during the last days and I already added support for relative paths locally. Also, I found a bug in my PR. It only affects exporting drumkits via the GUI but not upgrading them using the CLI. I will provide a patch for branch drumkit-cli-1.1.1 soon regardlessly (which will also contain the relative path support).
So, this will happen locally on the user's devices and you intend to release the patched |
|
@sten0 are there still some open points or is everything working fine with the CLI solution we came up with? |
|
theGreatWhiteShark ***@***.***> writes:
@sten0 are there still some open points or is everything working fine with the CLI solution we came up with?
Thank you for following up! Sadly while I was waiting for the free time
to implement the solution (trigger system drumkit xml file upgrade on
system drumkit package and/or Hydrogen package installation or upgrade),
an RC bug blocked all further progress. That was #1505
I'm currently waiting for a few rounds of arm64 CI results, and this
should take anywhere from three days to a week.
Please let me know if your drumkit upgrade branch will definitely be
merged into the next stable release. If it is, and if I'm able to find
free time, then I may be able to get a head start on completing this
work.
Oh, status update is that 1.1.1+cherry picked commits from your branch
builds, and I haven't yet implemented the install/upgrade trigger for
batch drumkit upgrade.
|
|
theGreatWhiteShark ***@***.***> writes:
@sten0 could you try #1564? If everything works fine, it will be part of the upcoming 1.1.3 release.
I've uploaded b917e05 from this branch, so user testing can commence.
I had to prioritise this because Hydrogen was in danger of being dropped
from Debian due to that release critical arm64 bug. Sadly all of my
free time went to resolving the delta between 1.1.1 release tarball and
a snapshot of the master branch, so I've had to defer implementation of
the trigger to batch-upgrade all drumkit.xml files.
By the way, is the new tutorial, in English only, the only available
documentation, or will the old docs be added back for 1.1.2?
|
If such a thing happens at some point in the future again: please feel free to ping us and to ask to resolve such conflicts. If we have some time at hand, we might help :)
The docs are actually cutting-edge right now. I rewrote them for the 1.1 release and they work for all patch releases in the 1.1.* series too. But the commit checked out in the submodule might not the the proper one. We also wanted to keep the tags in the hydrogen repo and those in the documentation consistent. But it seems we didn't 😁 . I'll have a look. The tutorial is a different story. It is as old as (git) history itself (at least 15 years). Maybe I find time to update it for 1.2. But right now it can be considered seriously out of date. |
Fixed |
|
>> Sadly all of my free time went to resolving the delta between 1.1.1
>> release tarball and a snapshot of the master branch, so I've had to
>> defer implementation of the trigger to batch-upgrade all drumkit.xml
>> files.
>
> If such a thing happens at some point in the future again: please feel free to ping us and to ask to resolve such conflicts. If we have some time at hand, we might help :)
>
Thanks! :-) In this case I thought it would be quick, and also felt I
should educate myself a bit about Hydrogen...I'm sure you've also lost
a lot of time to a "I refuse to be defeated" mood haha!
>> By the way, is the new tutorial, in English only, the only available
>> documentation, or will the old docs be added back for 1.1.2?
>
> The docs are actually cutting-edge right now. I rewrote them for the
> 1.1 release and they work for all patch releases in the 1.1.* series
> too.
Oh wow, that's great news :-) Have they also been translated, or would
you like me to ask if anyone on the Debian translation teams might be
interested?
> But the commit checked out in the submodule might not the the
> proper one. We also wanted to keep the tags in the **hydrogen** repo
> and those in the **documentation** consistent. But it seems we didn't
> 😁 . I'll have a look.
>
Fixed
Thanks, much appreciated! :-)
The tutorial is a different story. It is as old as (git) history
itself (at least 15 years). Maybe I find time to update it for
1.2. But right now it can be considered seriously out of date.
Mm, I noticed that too. Would you say this is a case where "some
translated documentation" is actually worse than no documentation? If
so, please let me know, and I'll drop it from the Debian package so as
not to mislead users.
In other news, it seems I've run out of free time again. Please
continue to ping me about implementing the drumkit.xml file upgrade
functionality.
|
Well, there were some translations at some point. But after the rewrite none of them reached 30% coverage. We discussed about dropping them (or moving them to some point they won't be installed anymore) but we haven't done it yet. Yes, that would be awesome. I'm not a 100% sure at which point to do so, though. I will apply some changes during the next month to keep everything up-to-date for the 1.2.0. Afterwards, the document will probably we stable for quite some time.
Hmm. Not sure. But I think it's still better than other stuff out there. E.g. there are tutorials on youtube working with version such as 0.7.5 which is almost 20 years old.
At least as far as I can see installation, extraction, validation, and upgrading is now working fine both with absolute and relative paths. Also, the code is in both |
Is this something that is delaying the 1.20 release? Ah, now I see, the documentation has its own separate project now. It sounds like it will be best to just create a new Debian package for the new docs :) For lots of reasons. I'm currently blocked by missing copyright and licence
Thanks for the ACK! I think at this point I'll need to delegate, because this Debian work will need to be complete in the next three months. |
No, not at all. I already did the update of the docs and they will be tag as soon as the 1.2.0 beta release of Hydrogen itself is done. The latter is, unfortunately, blocked due to me. Apart from all the visual improvement the main change in the upcoming version is a rewrite of large parts of Hydrogen's audio engine to get rid off long standing glitches, bugs, and inconsistencies. But recently we found that there are still occasional glitches. I'm working to kill them and already covered a lot of ground. But I do not want to make a release without those fixes because consistency in the audio transport and rendering is so essential it will be worth the wait (at least I hope so).
At least as far as git history is concerned the docs were always separated from the main repo. By the way: it is possible access a local copy of the docs from within Hydrogen using Info > User Manual. So, the docs need to be a dependency of the main repo. |
|
Hey @sten0, CLI options to upgrade kits are now released as part of version 1.2.0. Are there still aspects of your issue that aren't covered yet or are we done for now? |
|
theGreatWhiteShark ***@***.***> writes:
Hey @sten0,
CLI options to upgrade kits are now released as part of version 1.2.0. Are there still aspects of your issue that aren't covered yet or are we done for now?
Hi @theGreatWhiteShark,
Thank you for following up on this issue. To be honest I'm not sure,
because I wasn't able to test yet. 1.2.0 was released during Debian 12
(bookworm)'s hard freeze
(https://release.debian.org/bullseye/freeze_policy.html), so it
couldn't/can't be merged into the upcoming Debian release. It could be
as late as July before I can do work for a proof-of-concept upload to
the "experimental" suite. In any case, no Hydrogen-new-release related
work will reach users until after bookworm is released, so I've been
spending my free time fixing various bugs (non-Hydrogen) that affect the
upcoming release. Another reason I'm not working on Hydrogen right away
is because this release will require another copyright review...just a
toilsome formality.
Once Debian 12 is released, and after Hydrogen 1.2.0 (final/release)
migrates to testing/trixie (future Debian 13), then I'll be willing to
do a backport of this Hydrogen release to Debian 12/bookworm--this will
just require someone other than myself to request a backport. In other
words, having missed the hard freeze deadline will inconvenience to
users, but they can still run an officially packaged and supported
latest-stable-version of Hydrogen with an official opt-in repository
(bullseye-backports) :)
|
I know. It's not ideal. But we still doing our best with limited resources and Debian users will get a hardened and more stable patch release of 1.2. right away. Then let's keep this one open for now. |
Hydrogen version * : 1.1.1
Operating system + version : Debian sid/unstable (will become 12 "Bookworm")
Audio driver + version : NA
Dear Hydrogen development team,
I'm the Debian maintainer for this software, and I noticed the following errors with the latest release:
Steps to reproduce:
Expected behaviour:
All drumkit.xml files should be "updated" when the package is built, and this seems like something that should be done by default in the upstream rather than worked around during build time in every distribution. Alternatively maybe these files shouldn't be installed at all, but should instead be generated in $HOME.
This also affects Ubuntu and its derivatives, and the deadline for getting this fix into Ubuntu 22.04 (and derivatives) LTS is 16 Feb, because it needs time to migrate through Debian's QA before Ubuntu does their last sync on 23 Feb.
The text was updated successfully, but these errors were encountered: