td-agent 4 #209

repeatedly · 2019-07-08T07:50:01Z

Update components:

ruby 2.7
latest fluentd v1
other bundled libraries, jemalloc or etc.

Drop components

Remove lsb-core dependency from systemd based OS

New support OS:

~~Debian 10~~(support in td-agent 3)
~~RHEL / CentOS 8 support (CentOS is not released yet)~~(support in td-agent3)
Ubuntu 20.04
Provide arm 64bit version for CentOS8/Amazon Linux 2/Ubuntu 18.04/Ubuntu 20.04/Debian 10

Drop OS:

CentOS 6 32bit
Debian 8
Ubuntu 12.04, 14.04

New plugin:

No idea for now

Drop plugin:

fluent-plugin-td-monitoring

td-agent 4 is based on new build system: fluent/fluent-package-builder#24

keuko · 2019-09-09T08:50:06Z

Update components:

ruby 2.6

latest fluentd v1

other bundled libraries, jemalloc or etc.

Drop components

Remove lsb-core dependency from systemd based OS

New support OS:

Debian 10

RHEL / CentOS 8 support (CentOS is not released yet)

Provide arm 64bit version

Drop OS:

CentOS 6 32bit

Debian 8

Ubuntu 12.04, 14.04

New plugin:

No idea for now

Drop plugin:

fluent-plugin-td-monitoring

First release date:

September or October

Dear Masahiro-san,

We are working on new openstack kolla release where we are using td-agent package ( https://review.opendev.org/#/c/680447/6/docker/base/sources.list.debian ). Currently we are working on new buster images and we are missing buster version of td-agent . I just want to ask if you can release it or when it will be possible to release.

Thanks,
Michal Arbet

repeatedly · 2019-09-11T11:33:18Z

I tested buster and building package is succeeded.
We will provide deb for buster in later this week.

keuko · 2019-09-17T13:38:54Z

Hello, could you please provide apt URL ?

viktormichalek · 2019-09-23T18:04:22Z

Dear Masahiro-san,

when the td-agent package for Debian Buster will be available?

My apologies for my impatience.
Viktor Michalek

repeatedly · 2019-09-24T23:04:42Z

I just uploaded 3.5.0 to buster. Please check it.

new openstack kolla release

If you need the package for your big project, we recommend to create own package for it like google-fluentd.

repeatedly · 2019-09-25T23:12:00Z

Note: ruby 2.6.4 is not good for fluentd. We need to use 2.6.3 or wait 2.6.5.

viktormichalek · 2019-09-30T15:53:44Z

Thank you Masahiro-san.

Thanks a lot to Masahiro Nakagawa treasure-data/omnibus-td-agent#209 (comment) Change-Id: I9c4993436844d45bc9df2815659fa7c420910b01

* Update kolla from branch 'master' - Merge "Fluentd: There is available td-agent pkg for Buster" - Fluentd: There is available td-agent pkg for Buster Thanks a lot to Masahiro Nakagawa treasure-data/omnibus-td-agent#209 (comment) Change-Id: I9c4993436844d45bc9df2815659fa7c420910b01

lmxia · 2019-10-22T07:43:45Z

Hi， i am using td-agent3 on ubuntu.
And ruby version2.4 is susceptible to vulnerabilities, which described in https://security.netapp.com/advisory/ntap-20190221-0002.

td-agent4 which has ruby version 2.6.3 is good for me. when is the td-agent4 available for us? i can't wait to get this. thanks a lot

repeatedly · 2019-10-22T11:32:58Z

And ruby version2.4 is susceptible to vulnerabilities,

Latest td-agent 3 uses ruby 2.4.6 so above security issue doesn't exist.

omnibus-td-agent/config/projects/td-agent3.rb

Line 22 in d2f46d1

override :ruby, :version => '2.4.6'

hrw · 2019-12-11T17:43:21Z

Any update? It is December now...

repeatedly · 2019-12-24T02:44:59Z

We have a plan to update bundled component to ruby 2.7 and fluentd v1.9.
So the release date will be delayed until the late of Jan.

repeatedly · 2019-12-24T02:46:09Z

I have built arm64 package for ubuntu 18.04 and amazon linux 2.

https://s3.amazonaws.com/packages.treasuredata.com/test/td-agent_3.5.1-0_arm64.deb
https://s3.amazonaws.com/packages.treasuredata.com/test/td-agent-3.5.1-0.amazon2.aarch64.rpm

If you are interesting in td-agent for Arm, please test it.

kevinzs2048 · 2020-01-07T06:58:53Z

@repeatedly Thanks very much

eredi93 · 2020-02-20T22:43:58Z

@repeatedly any update on the release of td-agent 4?
i'm currently blocked due to a limitation of ruby 2.4.x #237

kevinzs2048 · 2020-03-09T08:27:39Z

@repeatedly Thanks! I build on Debian Buster Arm64 and tested, it works.

hrw · 2020-03-12T10:57:10Z

It is March now. I doubt that it ever get released.

Looks like you are unable to manage it. 3.6.0 got Amazon Linux package for aarch64. But not for other distributions which suggests that someone built it by hand. While nowadays such things are done by CI/CD systems.

Treasure-data is part of Arm but you are unable to publish packages for Arm platforms for quite a while (I prefer not to check when you were asked to build them).

Shame.

repeatedly · 2020-03-16T07:10:33Z

We are now working on new build system with clearcode team: https://github.com/clear-code/td-agent-builder
This will resolve several omnibus issues: #219

td-agent 4 uses this td-agent-builder. We have a plan to release td-agent 4 RC in the latter of this month.

hrw · 2020-04-30T15:11:33Z

How work goes? CentOS 8 will not ship fluentd so you have to speed up a bit :D

repeatedly · 2020-05-09T05:33:24Z

https://td-agent-package-browser.herokuapp.com/test

We put td-agent 4.0.0 rc1 for Ubuntu/CentOS8. We appreciate any feedback.
We will reduce package size by removing unnecessary files.

Base branch is here: fluent/fluent-package-builder#24

hrw · 2020-05-10T16:26:35Z

RPM package look non-striped compared to Debian one - 62MB difference in /opt/td-agent/lib/ files.

Add striping as current RPM package enlarges image without any reason.

repeatedly · 2020-06-03T14:47:40Z

After fixed nokogiri issue for windows, we have a plan to release td-agent 4
rc1 packages for centos8/ubuntu 18.04/ubuntu 20.04 are uploaded

hrw · 2020-06-03T15:07:50Z

So when we can expect working repositories?

centos 8
debian buster
ubuntu bionic
ubuntu focal

Of course aarch64 and x86-64 ones.

Xphobia · 2020-06-05T03:53:44Z

where is working repositories for ubuntu/aarch64 platform? or any way to build by myself ? i've checked official repo, only amd64 .deb availablable.

hrw · 2020-07-02T09:24:39Z

Can someone close this issue as WONTFIX/KTHXBYE/INVALID?

It would summarize authors attempts to get td-agent 4 built and distributed.

repeatedly · 2020-07-02T11:58:48Z

OK.

Working on td-agent 4 release: fluent/fluent-package-builder#123

eredi93 · 2020-07-08T17:17:34Z

@repeatedly do you know when it'll be available via http://packages.treasuredata.com YUM repo?
(i'm using AMZ linux)

repeatedly · 2020-07-08T21:41:25Z

We don't provide amazon linux package. Use amazon linux 2.

https://td-agent-package-browser.herokuapp.com/4/amazon/2

eredi93 · 2020-07-08T22:06:18Z

@repeatedly i'm using amazon linux 2. i just found what the issue was.

https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent3.sh

as a base url of baseurl=http://packages.treasuredata.com/3/amazon/2/\$releasever/\$basearch

i tried to update the 3 with 4 but it was erroring with 404s

it seems that the problem is the extra /2/

baseurl=http://packages.treasuredata.com/4/amazon/\$releasever/\$basearch worked 👌

[ec2-user@my-host ~]$ sudo yum update td-agent
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
194 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package td-agent.x86_64 0:3.7.1-0.amazon2 will be updated
---> Package td-agent.x86_64 0:4.0.0-1.amzn2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================
 Package                            Arch                             Version                                   Repository                              Size
============================================================================================================================================================
Updating:
 td-agent                           x86_64                           4.0.0-1.amzn2                             treasuredata                            13 M

Transaction Summary
============================================================================================================================================================
Upgrade  1 Package

Total download size: 13 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
td-agent-4.0.0-1.amzn2.x86_64.rpm                                                                                                    |  13 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : td-agent-4.0.0-1.amzn2.x86_64                                                                                                            1/2
warning: /etc/td-agent/td-agent.conf created as /etc/td-agent/td-agent.conf.rpmnew
Warning: td-agent.service changed on disk. Run 'systemctl daemon-reload' to reload units.
prelink detected. Installing /etc/prelink.conf.d/td-agent-ruby.conf ...
old-prelink detected, Update /etc/prelink.conf ...
  Cleanup    : td-agent-3.7.1-0.amazon2.x86_64                                                                                                          2/2
  Verifying  : td-agent-4.0.0-1.amzn2.x86_64                                                                                                            1/2
  Verifying  : td-agent-3.7.1-0.amazon2.x86_64                                                                                                          2/2

Updated:
  td-agent.x86_64 0:4.0.0-1.amzn2

Complete!
[ec2-user@my-host ~]$ sudo yum info td-agent
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
194 packages excluded due to repository priority protections
Installed Packages
Name        : td-agent
Arch        : x86_64
Version     : 4.0.0
Release     : 1.amzn2
Size        : 56 M
Repo        : installed
From repo   : treasuredata
Summary     : The stable distribution of Fluentd
URL         : https://www.treasuredata.com/
License     : ASL 2.0
Description : The stable distribution of Fluentd, called td-agent.

[ec2-user@my-host ~]$

This was referenced Aug 13, 2019

Provide builds for AArch64 and PPC64LE architectures #131

Open

No binary package for Debian 10 'buster' fluent/fluentd#2571

Closed

openstack-gerrit pushed a commit to openstack/kolla that referenced this issue Oct 2, 2019

Fluentd: There is available td-agent pkg for Buster

4b30869

Thanks a lot to Masahiro Nakagawa treasure-data/omnibus-td-agent#209 (comment) Change-Id: I9c4993436844d45bc9df2815659fa7c420910b01

vwbusguy mentioned this issue Oct 10, 2019

Support CentOS 8 #205

Closed

repeatedly mentioned this issue Jan 6, 2020

arm64 package build (Ubuntu bionic deb) fluent/fluentd#2752

Closed

eredi93 mentioned this issue Feb 20, 2020

change ruby version? #237

Closed

alazyer mentioned this issue Mar 25, 2020

Docker 多架构构建 alazyer/mdblog#24

Closed

ashie mentioned this issue Mar 30, 2020

Time to goodbye omnibus? #219

Closed

cosmo0920 mentioned this issue Apr 7, 2020

Bump up using Ruby version to 2.6 fluent/fluent-package-builder#23

Closed

repeatedly closed this as completed Jul 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

td-agent 4 #209

td-agent 4 #209

repeatedly commented Jul 8, 2019 •

edited

Loading

keuko commented Sep 9, 2019

repeatedly commented Sep 11, 2019

keuko commented Sep 17, 2019

viktormichalek commented Sep 23, 2019

repeatedly commented Sep 24, 2019

repeatedly commented Sep 25, 2019

viktormichalek commented Sep 30, 2019

lmxia commented Oct 22, 2019

repeatedly commented Oct 22, 2019

hrw commented Dec 11, 2019

repeatedly commented Dec 24, 2019

repeatedly commented Dec 24, 2019 •

edited

Loading

kevinzs2048 commented Jan 7, 2020

eredi93 commented Feb 20, 2020

kevinzs2048 commented Mar 9, 2020

hrw commented Mar 12, 2020 •

edited

Loading

repeatedly commented Mar 16, 2020

hrw commented Apr 30, 2020

repeatedly commented May 9, 2020 •

edited

Loading

hrw commented May 10, 2020

repeatedly commented Jun 3, 2020 •

edited

Loading

hrw commented Jun 3, 2020

Xphobia commented Jun 5, 2020

hrw commented Jul 2, 2020

repeatedly commented Jul 2, 2020

eredi93 commented Jul 8, 2020

repeatedly commented Jul 8, 2020 •

edited

Loading

eredi93 commented Jul 8, 2020

td-agent 4 #209

td-agent 4 #209

Comments

repeatedly commented Jul 8, 2019 • edited Loading

keuko commented Sep 9, 2019

repeatedly commented Sep 11, 2019

keuko commented Sep 17, 2019

viktormichalek commented Sep 23, 2019

repeatedly commented Sep 24, 2019

repeatedly commented Sep 25, 2019

viktormichalek commented Sep 30, 2019

lmxia commented Oct 22, 2019

repeatedly commented Oct 22, 2019

hrw commented Dec 11, 2019

repeatedly commented Dec 24, 2019

repeatedly commented Dec 24, 2019 • edited Loading

kevinzs2048 commented Jan 7, 2020

eredi93 commented Feb 20, 2020

kevinzs2048 commented Mar 9, 2020

hrw commented Mar 12, 2020 • edited Loading

repeatedly commented Mar 16, 2020

hrw commented Apr 30, 2020

repeatedly commented May 9, 2020 • edited Loading

hrw commented May 10, 2020

repeatedly commented Jun 3, 2020 • edited Loading

hrw commented Jun 3, 2020

Xphobia commented Jun 5, 2020

hrw commented Jul 2, 2020

repeatedly commented Jul 2, 2020

eredi93 commented Jul 8, 2020

repeatedly commented Jul 8, 2020 • edited Loading

eredi93 commented Jul 8, 2020

repeatedly commented Jul 8, 2019 •

edited

Loading

repeatedly commented Dec 24, 2019 •

edited

Loading

hrw commented Mar 12, 2020 •

edited

Loading

repeatedly commented May 9, 2020 •

edited

Loading

repeatedly commented Jun 3, 2020 •

edited

Loading

repeatedly commented Jul 8, 2020 •

edited

Loading