New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Including CasparCG server in Debian? #1104

Closed
petterreinholdtsen opened this Issue Nov 6, 2018 · 57 comments

Comments

9 participants
@petterreinholdtsen
Copy link
Contributor

petterreinholdtsen commented Nov 6, 2018

Hi

Are you interested in having CasparCG included as an official package in Debian and derivatives like Ubuntu? I've created a request for a CasparCG package in Debian (aka a WNPP request), and thought it best to check with you if this is something you welcome and would like to help make happen.

I am a Debian developer and can assist. It is a lot easier to do when the Debian maintainer and the upstream developers cooperate, so I find it best to ask up front if you are interested.

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 3, 2018

Anyone had time to look at this yet?

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 3, 2018

I tried to build the current code on Debian Buster/testing, and while most dependencies seem to be available, at least one piece is missing: Chromium Embedded Framework. I've requested this too into Debian just now.

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 3, 2018

The draft build rules for Debian is now available from https://github.com/Frikanalen/casparcg-server/tree/debian/debian .
The code disable the HTML module because CEF is missing, and adjust cmake rules to accept boost versions after 1.66 too.

@walterav1984

This comment has been minimized.

Copy link
Contributor

walterav1984 commented Dec 4, 2018

Thanks for requesting a official CEF package for Debian.

Another Debian repo package which also rely(additional features) on CEF but doesn't include it is called Nageru a live videomixer. The author and if I'm not mistaking maintainer of the Debian package @sesse has included notes to build the nageru with CEF in its readme.

My guess for not directly including/shipping is the License of CEF (new BSD) vs Nageru (GPLv3) / CasparCG(GPLv3). However I see that Debian allows (modified BSD license) in main repo but I'm not sure if "modified" is the same as "new".

Would it be a option/possibility to submit a official CEF build (independent) Debian package that both CasperCG and Nageru can link with so they don't have to ship with conflicting licenses themselves?

Or is a CEF framework build so application specific that its impossible to ship a common build/module that more applications can use?

Is this somewhat similar in the License/module issue/solution like BMD Decklink / AJA -drivers perhaps Newtek NDI option?

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 4, 2018

Thanks for doing this, I know there is some interest within the wider community for this, but I am not sure if there is from SVT currently.
One thing to note is that linux support is still not a supported way of running, but I believe that is intended to change in 2.3 or 3.0

One concern I have with this is that development resources are very limited and so progress is slow and supporting multiple distros/versions will potentially add more work.

As for ffmpeg and CEF dependencies, the only potential problem I can think of with using system versions is that it limits the version that can be used.
For something like CEF, I expect we will want to consider updating the version used again in the next version, and so using a system version will block that from happening unless the system will offer multiple versions. FFmpeg is less of an issue, but might face something similar if there is an important change in support of a common broadcast codec.

The current method of dependencies while not perfect works well as we can control exactly what versions we use, and ensure they are consistent across all supported platforms.

@sesse

This comment has been minimized.

Copy link
Contributor

sesse commented Dec 4, 2018

Hi,

I'm not sure why you pull in licensing here. CEF is under a free license that is acceptable for inclusion in Debian and also compatible with GPLv3.

The primary reason why CEF is not in the Debian archive is security support. Chromium is a big and complicated package to support, and it's out of the question to have an extra bundled copy of it within a CEF package. CEF upstream only supports the latter (and really only supports building by pulling live git snapshots out of Bitbucket—the source tarballs available for download are explicitly not supported), but I've managed to package everything up so that one can build CEF against Debian's version of Chromium. I have some packages at https://storage.sesse.net/cef/ that demonstrate this, but they're without security support.

However, this requires the Chromium source code to be available in Debian package form, and it requires that CEF be rebuilt and/or upgraded whenever Chromium gets a security update. So far, nobody has been able to give me a definite yes or no on whether this would be acceptable—it comes down to that the Chromium maintainer is already overworked, and that this kind of rippling security updates is not currently well supported in the archive.

See Debian bug #893448 for more discussion.

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 4, 2018

I take it from the comments here (as well as the handling of the Debian related pull requests #1116, #1117, #1120 and #1121 as well as issues #1118 #1119) that there is interest to have casparcg in Debian.

I'll see if there is interest in the Debian multimedia team to maintain it within that framework.

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 4, 2018

Btw, as the Debian release freeze is quickly approaching, it would be useful to know when you plan to move 2.2 out of beta and make a new release. Do you have a timeline for it? Debian freezes in a few weeks, most likely early February 2019. I would like to use a source version with clear copyright information, to reduce the chance of having the package rejected by the ftpmasters, and version 2.0.7 (the last non-beta release tagged in git) is filled with embedded copies of libraries that are going to raise questions from the Debian ftpmasters.

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 4, 2018

Thanks for that explanation @sesse. I assume that means we are also unable to include libcef.so as part of our package?

I take it from the comments here (as well as the handling of the Debian related pull requests #1116, #1117, #1120 and #1121 as well as issues #1118 #1119) that there is interest to have casparcg in Debian.
I believe the general policy here at the moment is that bug fixes will be happily merged in, so I would say the handling of the other issues is very conclusive.

@dotarmin Is there an eta for the release of 2.2?
You can rule out trying to use 2.0.7, as it does not have any support for linux.

@sesse

This comment has been minimized.

Copy link
Contributor

sesse commented Dec 4, 2018

Thanks for that explanation @sesse. I assume that means we are also unable to include libcef.so as part of our package?

In what form? Compiling all of CEF yourself as part of the CasparCG build? Using the binary builds provided by CEF? How would you provide security support for such a package when Chromium needs upgrades? (The latter would certainly be unworkable for policy, license and portability reasons.) Or are you thinking of something else entirely?

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 4, 2018

In what form? Compiling all of CEF yourself as part of the CasparCG build? Using the binary builds provided by CEF? How would you provide security support for such a package when Chromium needs upgrades? (The latter would certainly be unworkable for policy, license and portability reasons.) Or are you thinking of something else entirely?

Either way really. Compiling CEF during the build would be a pretty horrible task, but if that is what has to be done, then it would be worth considering. I would prefer to use the binaries, but I can understand that doesn't conform to debian policy.
I don't expect we would provide provide security updates, which I do also expect to be a problem with debian. I certainly wouldn't want the major version of chromium to increase unexpectedly. I have had websites break in small ways from chrome updates, so the potential for that happening to caspar graphics would encourage me to never update debian

@sesse

This comment has been minimized.

Copy link
Contributor

sesse commented Dec 4, 2018

So there's two sides to the story here: Making a working Debian package, and having it in Debian proper. I assume pere wants the latter.

Compiling CEF yourself and putting it into the package (or in a separate package in the same repository) would be possible for the former. You would get a package that nobody cares about security-wise, but perhaps for most Caspar use case, it doesn't matter as you're likely to run trusted source code and never access the network. Shipping a Caspar package with a private libcef.so is unlikely to be accepted for Debian, but again, maybe it's possible to work with the Chromium maintainer and the security team to find a more official solution that also Nageru and OBS could depend on.

Using the prebuilt binaries would not confirm to GPL (you need to include the source and build scripts for everything you link against), so that's not an option even if you don't want to stay in Debian proper.

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 7, 2018

Steinar is right, I am interested in uploading the package to Debian for inclusion in the official package repository there.

And for the record, my plan is to upload as soon as a new release is tagged including the updated copyright statements currently in HEAD.

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 7, 2018

Thanks for the explanations @sesse.
This is something to discuss upstream with Debian then, but its not a blocker so that isn't urgent.

@petterreinholdtsen Could you add some instruction to BUILDING.md on how to build this deb locally? That would be useful to ensure it doesn't get broken in future (and means I can try it out)

@sesse

This comment has been minimized.

Copy link
Contributor

sesse commented Dec 7, 2018

@petterreinholdtsen: So you intend to upload a version without support for CEF (nor presumably Flash)? I'm not sure how useful that would be.

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 7, 2018

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 7, 2018

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 14, 2018

Are there any plans to make a new beta release before Christmas?

@dotarmin

This comment has been minimized.

Copy link
Member

dotarmin commented Dec 14, 2018

Are there any plans to make a new beta release before Christmas?

Main goal is to release a stable before Christmas. Otherwise a RC will be released before Christmas.

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 14, 2018

@petterreinholdtsen Awesome plan, I've worked about with debian packaging before so I know the technical side of it. I started to prepare a bit to get CasparCG to be more package-able. If/when I can get the MacOS version sorted, then I wish to have it inside of 'homebrew'!

Personally I'm only really interested in getting it packaged etc, bringing it into debian proper is an obvious goal, but dealing with red-tape/licences etc...I'm very glad if someone else can ensure we're doing the right thing there!

I'm going to read through this topic a bit more and will check out your commits to get a better idea where we are at the moment with it.

Do you have some tips on how we can make development easier? I'm using Ubuntu Bionic on my laptop, and have mostly compiled against that, of course we can build in Docker, but I prefer to build direct on my laptop when doing development work.

Does the Debian project have some build system we can use, I have access to plenty of server power, even at home, but it's nice if there is some official servers we can get the builds made on.

Are you on the CasparCG Slack channel? It could be nice to chat whilst fixing some thing.

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 14, 2018

Do you feel CMake is good still for the build scripts, is there one that integrates nicer with debian packaging, I didn't really look into that side of things yet? I'd like to enhance the build scripts anyway, Of course a lot of people want HTML templates, but it could be nice to be able to easily build without some features, depending on licences, it'd be nice even if we can have the modules in separate packages (as long as it doesn't affect performance).

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 14, 2018

Another thought: I'd prefer to have the build scripts outside of this repository, what do you think? I think it's cleaner to have just the source for CasparCG inside one repo etc...

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 14, 2018

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 14, 2018

Ok, what I meant with the development, was that I use Ubuntu Bionic on the laptop, mostly due to driver issues. So it's hard to compile for debian directly on this machine for the obvious deps. reasons. Not so important though, maybe I can switch to debian stable.

I'll try to find you on IRC later....

I might have been a bit unclear, it's not the build scripts (cmake) that I think should be in another repo, but the packaging scripts. Homebrew/apt/rpm/windoze....

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 14, 2018

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 14, 2018

I know a little about chroot...have used it to build some firmware's for routers etc. But I didn't know so much about using it against different releases. Thanks for that info!

@dotarmin @5opr4ni What's your opinion on having a new repository to just have the packaging scripts in? Probably the build server scripts (Jenkins) could live there also.

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 14, 2018

When I have needed to build across distros I use a temporary docker container. Nice and quick to get running and easy to optionally throw away straight after, but chroot also sounds like a good way.

In general I'm not keen on the packaging being outside of the repo, but it depends on who is intended to look after it. It sounds like it makes sense for debian to be done that way, but for producing a standard standalone windows/linux/mac build I think it best to keep it in the repo

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 14, 2018

My thoughts/reasons behind this is:
If you want to develop CasparCG you will clone this repo, and follow the instructions for setting up a development environment. So you won't care about packaging
If you want to package CasparCG, apart from distro maintainers, it's most likely just SVT that would want to modify those scripts for the 'official builds', or maybe some other broadcasters if they have some server management tools for example.
If you just want to use CasparCG then you just download the prebuilt images/packages etc etc.

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 14, 2018

I often find myself packaging a build locally so that I can send it to others for testing.
I think it would be quite common that any developer who makes a custom build to want to do the same.
On windows especially the build output produced by visual studio without packaging is a complete mess with some intermediate, junk and duplicate files so isnt great to have to distribute. Linux produces the correct and clean structure, so I am fine with nothing there.
So as long as for the current platform/distro you are running its only a couple of simple commands or there are scripts then I am happy.

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 14, 2018

Basically...if you wanted to do a local package...it'd be just a case of cloning that repo, and running the script to build for your system.

We already did a lot of cleanup etc, but it's certainly possible to clean up and streamline the repo, packages even more.

My thought anyway, is that the source repo would output a very clean folder ready for distro packaging, or even to just zip up. The packaging scripts would be more about adding installation scripts, dependency stuff and making sure things are put in the right destination folders according to the distro/OS.

@dotarmin

This comment has been minimized.

Copy link
Member

dotarmin commented Dec 15, 2018

@k0d: Another thought: I'd prefer to have the build scripts outside of this repository, what do you think? I think it's cleaner to have just the source for CasparCG inside one repo etc...

I totally disagree on this one actually (even if I also thought about same thing as you). The build-scripts should be bundled with the repository it-self. The repository should be self contained and the server should be buildable without any extra forking etc.

The CasparCG project has a lot of various users and all are not experts. Even if this is natural having various levels of knowledge we should still aim to make it as simple as possible for the community members and users of CasparCG projects to build the server, even if we provide auto builds.

:)

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 15, 2018

@dotarmin I didn't mean the build scripts....but the packaging and jenkins (build-server) scripts.

I've thought about this a lot, in some ways I still feel having them in one repo is great, but I often find that it's confusing when cloning a new project and seeing so many folders for packaging and different people's editor configs etc. Look at this repo for example https://github.com/nodejs/node, where would one even start to develop on that project :) And that's a project that I feel is actually quite clean compared to others!

If we have them in one repo, then I suggest a folder structure similar to this...
./src (the actual c++/c code)
./tools (scripts to do with building, packaging etc)
./Makefile (this is just a suggestion, but if we can have it so people can just "git clone ... && cd ...&& make" it's so nice and simple)
./resources (files used by the build/packaging scripts that are 'data' not 'scripts', such as the debian control/rules files, app logos...)

That way, a developer can easily clone, build and use the server.

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 15, 2018

The problem with trying to tidy it further is the number of files that expect to be at the root of the repo.
But what is the problem with having packaging scripts hidden away under tools?

I think we should aim to have build-scripts that produce a folder ready for zipping as part of this repo. Perhaps the scripts could even default to zipping it up, but with a param to skip doing that. From what I can tell, the default packaging step will be solely to zip it up, with mostly only build servers wanting to do something else first.

And I think it is reasonable to assume you are running it on the os/distro/version you want to build for, so the building in docker could be replaced with some simpler script.

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 15, 2018

I don't want to sound grumpy, but I have mentioned a few times now, build-scripts should be in this repo, packaging scripts (could) be in a separate one.

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 15, 2018

Sorry, my last comment was meant to be more along the lines of I agree build-scripts should be here. But if done correctly, then default packaging is a single extra command, so why not optionally do that too, so I don't know what the other repo would contain

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 15, 2018

It's purely to keep the main source repo from being polluted with code that's specific to certain distros.

My way of doing things is to hide platform specific code if possible. I hate to see code with so many #ifdef's etc in them. It's just so confusing. Of course, I'm not saying we change the source code, just explaining my way of thinking.

I ❤️Clean Code

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 16, 2018

I decided to reserve a slot in the Debian NEW queue for casparcg-server,
<URL: https://ftp-master.debian.org/new/casparcg-server_2.2.0~beta7%2Bdfsg-1.html >.

@k0d

This comment has been minimized.

Copy link
Collaborator

k0d commented Dec 16, 2018

How well does it work? can it find the config file for example? what do we need to do to get it ”finished”.

Shall we create a new milestone and create issues under that for this ’debianise casparcg’ project?

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 16, 2018

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 24, 2018

Main goal is to release a stable before Christmas. Otherwise a RC will be released before Christmas.

Very good to know. It would be useful for my own planning to know at what point in time the 'Christmas' you refer to occur. Can you provide some details on date and year? :)

@dotarmin

This comment has been minimized.

Copy link
Member

dotarmin commented Dec 24, 2018

We found a bug yesterday that Julian has fixed in branch 2.2.x (thanks Julian) but it also needs to be fixed in master branch. Estimated date is somewhere between 27-29 December.

Merry Christmas 🎅

@Julusian

This comment has been minimized.

Copy link
Member

Julusian commented Dec 24, 2018

That fix not being in master doesn't stop 2.2 being released, as master is 2.3/3.0, but the fix is merged to master now too

@dotarmin

This comment has been minimized.

Copy link
Member

dotarmin commented Dec 24, 2018

That fix not being in master doesn't stop 2.2 being released

Well, I use master in production as is now and that’s a fix needed related to mxf files in my case 😊 But it’s fixed now in master thanks to Julian 😍

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Dec 30, 2018

I uploaded version 2.2.0-stable to Debian a few minutes ago. The package has not yet passed NEW processing, so it is not available in Debian yet. No idea if the ftpmasters will find time to look at it before Debian freeze the next stable release, nor if it will propagate into the next stable release in time for the freeze. It seem quite unlikely at the moment, as there are few days left.

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Jan 14, 2019

I am happy to report that CasparCG server just was accepted into Debian, <URL: https://tracker.debian.org/pkg/casparcg-server >.

@5opr4ni

This comment has been minimized.

Copy link
Contributor

5opr4ni commented Jan 14, 2019

Super BIG smiley. Nice work.

@Zebiolo

This comment has been minimized.

Copy link

Zebiolo commented Jan 14, 2019

Freaking awesome!

@5opr4ni: It is time for SVT to roll out Debian on your servers :)

@dimitry-ishenko

This comment has been minimized.

Copy link
Contributor

dimitry-ishenko commented Jan 14, 2019

Doesn't have to be Debian. Can do Ubuntu too 😝

@sesse

This comment has been minimized.

Copy link
Contributor

sesse commented Jan 14, 2019

@5opr4ni: It is time for SVT to roll out Debian on your servers :)

That would seem difficult without any support for Flash nor CEF.

@dimitry-ishenko

This comment has been minimized.

Copy link
Contributor

dimitry-ishenko commented Jan 14, 2019

Oh, is cef not supported on 2.2?...

@sesse

This comment has been minimized.

Copy link
Contributor

sesse commented Jan 14, 2019

Oh, is cef not supported on 2.2?...

Debian ships CasparCG compiled without CEF support.

@5opr4ni

This comment has been minimized.

Copy link
Contributor

5opr4ni commented Jan 14, 2019

Yes can be a problem for CaparCG when used as a CG machine, but we also have more than 17 controlrooms using at least one CasparCG per gallery for only videoplayout.

@dotarmin

This comment has been minimized.

Copy link
Member

dotarmin commented Jan 15, 2019

This is just fantastic!

@petterreinholdtsen, the CasparCG community is really thankful for the job you have done to make this happened! Thanks! 🥇

@petterreinholdtsen

This comment has been minimized.

Copy link
Contributor

petterreinholdtsen commented Jan 15, 2019

@sesse

This comment has been minimized.

Copy link
Contributor

sesse commented Jan 15, 2019

The library is called -lpthread, not -lpthreads. (There's also the -pthread flag.)

@dotarmin

This comment has been minimized.

Copy link
Member

dotarmin commented Jan 15, 2019

Right now, a bug report
about a build problem intrigues me, as it fail because pthreads is
missing, and I fail to understand how that is even possible... :)

Somebody that can help @petterreinholdtsen with this issue?

Best regards,
Armin

@dotarmin

This comment has been minimized.

Copy link
Member

dotarmin commented Jan 15, 2019

The library is called -lpthread, not -lpthreads. (There's also the -pthread flag.)

Thanks @sesse 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment