Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

revert apriltag release #21437

Merged
merged 2 commits into from
Jun 3, 2019
Merged

revert apriltag release #21437

merged 2 commits into from
Jun 3, 2019

Conversation

tfoote
Copy link
Member

@tfoote tfoote commented Jun 3, 2019

It's not building on any platform and is taking many hours to fail

Reverting #21256 @wxmerkt

image

Hours of elapsed time per compilation unit on armhf

09:07:43.571 [ 21%] Building C object CMakeFiles/apriltag.dir/tagCircle21h7.c.o
09:07:43.577 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagCircle21h7.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagCircle21h7.c
09:07:44.651 [ 24%] Building C object CMakeFiles/apriltag.dir/tag36h11.c.o
09:07:44.657 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tag36h11.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tag36h11.c
09:07:46.804 [ 27%] Building C object CMakeFiles/apriltag.dir/tagStandard41h12.c.o
09:07:46.810 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagStandard41h12.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagStandard41h12.c
09:07:59.314 [ 30%] Building C object CMakeFiles/apriltag.dir/tagCustom48h12.c.o
09:07:59.320 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagCustom48h12.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagCustom48h12.c
11:31:24.696 [ 33%] Building C object CMakeFiles/apriltag.dir/tag16h5.c.o
11:31:24.702 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tag16h5.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tag16h5.c
11:31:25.892 [ 36%] Building C object CMakeFiles/apriltag.dir/common/time_util.c.o
11:31:25.898 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/common/time_util.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common/time_util.c
11:31:27.240 [ 39%] Building C object CMakeFiles/apriltag.dir/common/image_u8.c.o
11:31:27.246 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/common/image_u8.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common/image_u8.c
11:31:30.571 [ 42%] Building C object CMakeFiles/apriltag.dir/common/zmaxheap.c.o
11:31:30.578 /usr/lib/ccache/arm-linux-gnueabihf-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/common/zmaxheap.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common/zmaxheap.c

Many minutes on amd64

00:04:12.016 [  9%] Building C object CMakeFiles/apriltag.dir/apriltag.c.o
00:04:12.016 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/apriltag.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/apriltag.c
00:04:12.578 [ 12%] Building C object CMakeFiles/apriltag.dir/tag25h9.c.o
00:04:12.579 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tag25h9.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tag25h9.c
00:04:12.666 [ 15%] Building C object CMakeFiles/apriltag.dir/tagCircle49h12.c.o
00:04:12.667 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagCircle49h12.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagCircle49h12.c
00:19:44.687 [ 18%] Building C object CMakeFiles/apriltag.dir/tagStandard52h13.c.o
00:19:44.688 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagStandard52h13.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagStandard52h13.c
00:27:26.357 [ 21%] Building C object CMakeFiles/apriltag.dir/tagCircle21h7.c.o
00:27:26.357 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagCircle21h7.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagCircle21h7.c
00:27:26.417 [ 24%] Building C object CMakeFiles/apriltag.dir/tag36h11.c.o
00:27:26.417 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tag36h11.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tag36h11.c
00:27:26.566 [ 27%] Building C object CMakeFiles/apriltag.dir/tagStandard41h12.c.o
00:27:26.566 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagStandard41h12.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagStandard41h12.c
00:27:27.498 [ 30%] Building C object CMakeFiles/apriltag.dir/tagCustom48h12.c.o
00:27:27.499 /usr/lib/ccache/x86_64-linux-gnu-gcc  -Dapriltag_EXPORTS -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/. -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0 -I/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/common -I/apriltag  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -o CMakeFiles/apriltag.dir/tagCustom48h12.c.o   -c /tmp/binarydeb/ros-kinetic-apriltag-3.1.0/tagCustom48h12.c

Build failure from dpkg-shlibdeps on all platforms

11:32:57.811 	dpkg-shlibdeps -Tdebian/ros-kinetic-apriltag.substvars -l/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/debian/ros-kinetic-apriltag//opt/ros/kinetic/lib/ debian/ros-kinetic-apriltag/opt/ros/kinetic/bin/apriltag_demo debian/ros-kinetic-apriltag/opt/ros/kinetic/lib/arm-linux-gnueabihf/libapriltag.so.3.1.0
11:32:58.834 dpkg-shlibdeps: error: couldn't find library libapriltag.so.3 needed by debian/ros-kinetic-apriltag/opt/ros/kinetic/bin/apriltag_demo (ELF format: 'elf32-littlearm-hfabi'; RPATH: '')
11:33:02.215 dpkg-shlibdeps: error: cannot continue due to the error above
11:33:02.215 Note: libraries are not searched in other binary packages that do not have any shlibs or symbols file.
11:33:02.215 To help dpkg-shlibdeps find private libraries, you might need to use -l.
11:33:02.236 dh_shlibdeps: dpkg-shlibdeps -Tdebian/ros-kinetic-apriltag.substvars -l/tmp/binarydeb/ros-kinetic-apriltag-3.1.0/debian/ros-kinetic-apriltag//opt/ros/kinetic/lib/ debian/ros-kinetic-apriltag/opt/ros/kinetic/bin/apriltag_demo debian/ros-kinetic-apriltag/opt/ros/kinetic/lib/arm-linux-gnueabihf/libapriltag.so.3.1.0 returned exit code 2
11:33:02.241 debian/rules:47: recipe for target 'override_dh_shlibdeps' failed
11:33:02.241 make[1]: *** [override_dh_shlibdeps] Error 2

@tfoote
revert apriltag release
ba59b8d 
It's not building on any platform and is taking many hours to fail
@tfoote
restore apriltags2_ros
d7928a0 
accidentally removed in last commit
@tfoote tfoote merged commit 8abea24 into tfoote-reroll Jun 3, 2019
@tfoote tfoote deleted the tfoote-revert-21265 branch June 3, 2019 05:47
@wxmerkt
Copy link
Contributor

wxmerkt commented Jun 3, 2019

Thank you for the heads-up. I tried to have a look at the logs for many days but cannot since the build.ros.org is now behind a registration wall, and after registration I can't see anything.

[I am also receiving build failure emails for many packages that have been released successfully before - which made me think it's a fluke]

@wxmerkt
Copy link
Contributor

wxmerkt commented Jun 3, 2019

I'd love to reproduce and get this fixed - but I cannot see any build.ros.org status pages (and the links now lead to pages behind logins, with missing the permission Overall/Read). How much memory are the virtual images using? [I'd also like to extract the cmake configuration command]

@wxmerkt wxmerkt mentioned this pull request Jun 3, 2019
Closed
@wxmerkt
Copy link
Contributor

wxmerkt commented Jun 3, 2019

I've run the Kinetic-amd64 build in a Docker and it worked just fine. Do you have the full log with the full configure/build command for me to reproduce? (or perhaps add my username - wxmerkt - to the read permissions for build.ros.org so that I can check it?)
Thank you :)

@rgreid
Copy link

rgreid commented Jun 3, 2019

@wxmerkt https://discourse.ros.org/t/security-issue-on-ros-build-farm/9342

@wxmerkt
Copy link
Contributor

wxmerkt commented Jun 3, 2019

Thank you for the pointer @rgreid - I was looking for answers on answers.ros.org and totally forgot to theck Discourse.

tfoote added a commit that referenced this pull request Jun 5, 2019
@tfoote
Cleanup for rebuild of indigo and kinetic (#21466)
3f4691e 
* roll back rtt to 2.9.0-2

Re: orocos-toolchain/rtt#299
Reverts #21232

* revert apriltag release (#21437)
It's not building on any platform and is taking many hours to fail

* remove pyros_setup

The release repo has been deleted.
 It was eol'd in #13177

* remove micros_rtt release

The release repo is 404 now.
@wxmerkt
Copy link
Contributor

wxmerkt commented Jun 8, 2019

@tfoote Now that the buildfarm is back up, we would like to take a look at getting the failures resolved. @mkrogius provided insights and a pointer (AprilRobotics/apriltag#16 (comment)). The functions that take long to compile with -O2 or higher have attributes to deactivate explicitly for them - it appears that does not work on the buildfarm. Am I missing something?

@mkrogius
Copy link

I've attempted to reproduce the issue by using the same compiler/compiler flags to compile the project. However for me, I can build the whole project in 24 seconds.

What compiler version is the build server using? I have:
arm-linux-gnueabihf-gcc (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04) 7.4.0

On my machine, compilation consumes a maximum of 643MB so I doubt this is a memory problem.

@mkrogius
Copy link

Ok, I can reproduce this issue with version 5.4.0 of the same ARM compiler. Version 5.4.0 of the x86 compiler is also slow.

@mkrogius
Copy link

The build is quick, except for when both the "-g" and "-O2" flags are used.

@mkrogius
Copy link

It turned out the issue was track-vars: https://stackoverflow.com/questions/2954473/variable-tracking-is-eating-my-compile-time

@tfoote
Copy link
Member Author

tfoote commented Jun 14, 2019

Interesting. Thanks for digging into it. I see @wxmerkt got a new release out #21554 I'll merge it after the sync tomorrow.

@j-rivero j-rivero mentioned this pull request Jul 26, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tfoote @wxmerkt @rgreid @mkrogius