Conversation
|
Thanks for the patch, @epikhinm . |
|
Nice, could you show me how are you testing? |
|
Basically, you need to first get packages built: And then deploy: If you hit any problem, feel free to post it on dev@bigtop.apache.org |
bigtop_toolchain/bin/puppetize.sh
Outdated
| dnf -y install hostname findutils curl sudo unzip wget puppet puppetlabs-stdlib | ||
| ;; | ||
| ubuntu-16.04) | ||
| ubuntu-1[6,8].04) |
There was a problem hiding this comment.
Would be better to create another entry for 18.04 as there are different packages to be installed.
There was a problem hiding this comment.
+1. Having another paragraph yield better readably.
| ubuntu-16.04) | ||
| ubuntu-1[6,8].04) | ||
| apt-get update | ||
| apt-get -y install wget curl sudo unzip puppet software-properties-common puppet-module-puppetlabs-apt puppet-module-puppetlabs-stdlib |
There was a problem hiding this comment.
On ubuntu-18.04 two extra packages need to be installed to get successful bigtop build:
rsync, dh-python
There was a problem hiding this comment.
Hmm, sounds strange. I've builded bigtop and some packages (hadoop, hive, hbase, and zookeeper) without these dependencies.
But ok, i'll add your opinion as a patch.
I also have small troubles with systemd on ubuntu-18.04, standard "sysvinit compatibility" doesn't work properly. In the current initd scripts we creating lock directories, pid files. Systemd expect than file have same pair owner-group as a main process, but it don't.
I wrote new template for systemd services, but doesn't debugged it yet.
Give me a couple of days for this patches :)
There was a problem hiding this comment.
I've read code of generating init.d scripts for all components, aaand it's not so easy to create new templating script for all systemd services, because ~30 services have a custom generate* methods.
…/epikhinm/bigtop into feature/add-support-ubuntu-18.04
…/epikhinm/bigtop into feature/add-support-ubuntu-18.04
…/epikhinm/bigtop into feature/add-support-ubuntu-18.04
epikhinm
left a comment
epikhinm
left a comment
There was a problem hiding this comment.
Hi folks!
I have a many problems with this pull request, and i don't think that i have a lot of time and energy for doing this by oneself.
TL;DR
Just before support ubuntu 18.04 we should do some major things, and before doing it we should reach an agreement:
- Support puppet 5, current version is 3.
- Rewrite templating and code generation of initrd scripts and support systemd services.
- Use aptly insteadof reprepro ?
Okey, now longread:
-
Ubuntu 18.04 supports only puppet 5, but in bigtop we are using puppet 3. I've tried to using manifests from third version, but puppet fifth have new keyword like
site, and some configs clashes with puppet 5. I fixed that, but, i don't know, maybe some problems are still there. Living with manifests for two versions is not good idea, so we need to choice one of them and migrate all code to newest deploy system. -
Ubuntu 18.04 have magic converting tool from sysvinit to systemd services, but it doesn't work with current initd scritps. I successfully tried to write some of systemd services, but this way have a couple of shades:
a) We should generate this services as we generate initd services, but the current way is a bit of weird. A lot of bash templating, and current "bash templates" a not usable for new systemd services, just because the same things in both ways are very different. Working with locks, pids, controlling aliveness of a process, controlling user and group. I've tried to change current bash templates, but the code looks ugly.
b) A lot of componenents. And each components have a lot of services. I just can't write a templating script for all of this scripts, because every service have own specific things in bash methods, and u cannot convert it automatically, only by reading your eyes.
c) Current initrd scripts have custom features likerollingupgradeandinitin hdfs-namenode. Systemd doesn't support that, and the right ways is -- create new systemd service only for that! For each custom feature and action -- new systemd service. A lot of services. -
This is not blocker issue, but the execution of
./gradlew apton builded components doesn't support*.ddebartifacts. Newest reprepro doesn't support this files, and the problem looks like this ros-infrastructure/buildfarm_deployment#186 and we have three ways:
a) skip these artifacts, weird way
b) use new version of reprepro not from ubuntu repositories, which is also weird
c) use tool like https://www.aptly.info, which is more supportable and powerful in features. -
Also not blocker, but the building of packages right now doesn't sign they. If you trying to run integration tests with packages from the repository on local filesystem, the default behavior for ubuntu is reject all unsigned packages. So, just installing components doesn't work with repository on filesystem. We should sign the packages, or, add option
allow_unsigned => trueinbigtop-deploy/puppet/manifests/bigtop_repo.ppand[trusted=yes]`` inprovisioner/utils/setup-env-debian.sh``` for trust unsigned artifacts.
If anyone have a time for support 18.04 it would be great for doing it together with splitting this pull request to sub tasks and discussion it.
So, anyone?:)
| ubuntu-16.04) | ||
| ubuntu-1[6,8].04) | ||
| apt-get update | ||
| apt-get -y install wget curl sudo unzip puppet software-properties-common puppet-module-puppetlabs-apt puppet-module-puppetlabs-stdlib |
There was a problem hiding this comment.
I've read code of generating init.d scripts for all components, aaand it's not so easy to create new templating script for all systemd services, because ~30 services have a custom generate* methods.
3 participants
Hi!
You are deleted support of ubuntu 14.04 LTS before end of life, soo i've added support ubuntu 18.04 LTS for amd64.
At ubuntu 18.04 we only have puppet 5.3 and we should add old directory /etc/puppet/modules as modulepath.
Pls, check this out. Feel free for feedback.