Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many tests fail on FreeBSD #231

Open
yurivict opened this issue Jul 21, 2017 · 45 comments
Open

Many tests fail on FreeBSD #231

yurivict opened this issue Jul 21, 2017 · 45 comments
Assignees
Labels
Milestone

Comments

@yurivict
Copy link

@yurivict yurivict commented Jul 21, 2017

26% tests passed, 84 tests failed out of 113

The following tests FAILED:
	 11 - publishing_single_path_resources_http_get_acceptance_test_suite (OTHER_FAULT)
	 12 - publishing_single_path_resources_http_put_acceptance_test_suite (OTHER_FAULT)
	 13 - publishing_single_path_resources_http_head_acceptance_test_suite (OTHER_FAULT)
	 14 - publishing_single_path_resources_http_post_acceptance_test_suite (OTHER_FAULT)
	 15 - publishing_single_path_resources_http_trace_acceptance_test_suite (OTHER_FAULT)
         ...

Example failures:

# ./work/restbed-4.6/test/regression/uncaught_exception_when_peer_closes_connection_regression_test_suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
uncaught_exception_when_peer_closes_connection_regression_test_suite is a Catch v1.9.6 host application.
Run with -? for options

-------------------------------------------------------------------------------
peer closes connection without sending data
-------------------------------------------------------------------------------
test/regression/source/uncaught_exception_when_peer_closes_connection.cpp:55
...............................................................................

test/regression/source/uncaught_exception_when_peer_closes_connection.cpp:55: FAILED:
due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

Abort trap
# ./work/restbed-4.6/test/regression/path_parameters_off_by_one_regression_test_suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
path_parameters_off_by_one_regression_test_suite is a Catch v1.9.6 host application.
Run with -? for options

-------------------------------------------------------------------------------
single path parameter with root are off by one
-------------------------------------------------------------------------------
test/regression/source/path_parameters_off_by_one.cpp:41
...............................................................................

test/regression/source/path_parameters_off_by_one.cpp:66: FAILED:
  REQUIRE( 200 == response->get_status_code( ) )
with expansion:
  200 == 0

test/regression/source/path_parameters_off_by_one.cpp:41: FAILED:
  {Unknown expression after the reported line}
with expansion:

due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 2 | 2 failed

Abort trap
@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 22, 2017

Please state your environment compiler, OS, etc...

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Jul 22, 2017

FreeBSD 11.1 amd64, clang-4.0.0.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 22, 2017

Thanks, attempting a build now.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Jul 22, 2017

Here's the port. It builds fine. If you can't reproduce on linux, the easiest is to build the port in the FreeBSD VM.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220912

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 23, 2017

So I've attempted to build on FreeBSD 11 AMD64 with clang-3.8.0 with no errors reported or tests failing.

Are you able to supply anymore information about the issues you see?

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Jul 23, 2017

I just reran build and tests with clang-3.8 and still the same.

Here is another failing case output:

# ./work/restbed-4.6/test/acceptance/custom_service_error_handling_acceptance_test_suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
custom_service_error_handling_acceptance_test_suite is a Catch v1.9.6 host application.
Run with -? for options

-------------------------------------------------------------------------------
Scenario: custom service error handler
     Given: I publish a faulty resources and set a service error handler
      When: I perform a HTTP 'GET' request to '/resources/1'
      Then: I should see a '0' (No Appropriate Status Message Found) status
            code
-------------------------------------------------------------------------------
test/acceptance/source/custom_error_handling/service.cpp:76
...............................................................................

test/acceptance/source/custom_error_handling/service.cpp:79: FAILED:
  REQUIRE( "No Appropriate Status Message Found" == response->get_status_message( ) )
with expansion:
  "No Appropriate Status Message Found"
  ==
  "Error"

test/acceptance/source/custom_error_handling/service.cpp:76: FAILED:
  {Unknown expression after the reported line}
with expansion:

due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 3 | 1 passed | 2 failed

Abort trap

I traced its syscalls with ktrace, here is a suspicious place:

 41704 custom_service_erro CALL  connect(0xa,0x7fffdfffcac0,0x10)
 41704 custom_service_erro STRU  struct sockaddr { AF_INET, 127.0.0.1:1984 }
 41704 custom_service_erro RET   connect -1 errno 36 Operation now in progress

The testcase attempts to connect socket with fd=0xa, gets "Operation now in progress", which means that it needs to try again later, and it never tries again as it should.

Are you sure custom_service_error_handling_acceptance_test_suite passes in all 3 cases for you?

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 23, 2017

I'm seeing green across the board. I'll upgrade my version of clang and see if I can't replicate the problems.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 28, 2017

Please provide the exact build arguments in use. I'm having a hell of time replicating this issue, which is alarming.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Jul 29, 2017

Here is how to reproduce:

  1. cd /usr/ports
  2. fetch https://bz-attachments.freebsd.org/attachment.cgi?id=184583 -o restbed.shar
  3. sh restbed.shar
  4. cd www/restbed
  5. make test
  6. observe test failures
@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 29, 2017

Thank you! I will test this scenario and give appropriate feedback in the next hour.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 29, 2017

Please note you mention the license is GPL, this is incorrect. We have adopted the AGPL.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 29, 2017

The archive you supplied fails with:

c - www/restbed
Ambiguous output redirect.
@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 29, 2017

Have you attempted to build the project as per the README instructions? If so, what was the result?

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Jul 29, 2017

Please note you mention the license is GPL, this is incorrect. We have adopted the AGPL.

Thanks, will correct it.

Ambiguous output redirect.

Such message should not occur from the above commands.

Have you attempted to build the project as per the README instructions? If so, what was the result?

Just tried, same result.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jul 29, 2017

Following the updated instructions results in:

The Corvusoft:asio-dependency:f5c5708:a/dependency/asio GH_TUPLE line has
a tag containing something else then [a-zA-Z0-9_]
*** Error code 1

Stop.
make: stopped in /usr/ports/www/restbed
@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Aug 3, 2017

Instructions as a shell script.
If fails for me with the messages from my OP.

#!/bin/sh

cd /usr/ports &&
rm -rf www/restbed
fetch https://bz-attachments.freebsd.org/attachment.cgi?id=184583 -o restbed.shar &&
sh restbed.shar &&
cd www/restbed &&
make test
@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 5, 2017

root@freebsd: ~ # ./install.sh
fetch: https://bz-attachments.freebsd.org/attachment.cgi?id=184583: size of remote file is not known
restbed.shar                                                            4209  B      10  kBps 00m00s
c - www/restbed
x - www/restbed/distinfo
x - www/restbed/Makefile
x - www/restbed/pkg-descr
x - www/restbed/pkg-plist
The Corvusoft:asio-dependency:f5c5708:a/dependency/asio GH_TUPLE line has a tag containing something else than [a-zA-Z0-9_]
*** Error code 1

Stop.
make: stopped in /usr/ports/www/restbed
root@freebsd: ~ # uname -a
FreeBSD freebsd 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
root@freebsd: ~ #
@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 5, 2017

Can I ask you to confirm the following?

git clone --recursive https://github.com/corvusoft/restbed.git
mkdir restbed/build
cd restbed/build
cmake -DBUILD_TESTS=YES -DBUILD_EXAMPLES=YES ..
make install
make test

If you receive errors please provide as much information as possible.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Aug 5, 2017

If you receive test errors please provide as much information as possible.

Exactly the same errors as before.

What's the version you are running? What does 'uname -a' say?

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 5, 2017

I provided the output from uname in my previous post.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 5, 2017

Odd I see no failures when running the README.md build instructions.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 5, 2017

I'm currently operating on VirtualBox. I'll move to hardware to ensure there is no funny business.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Aug 5, 2017

It's odd that you get this:

The Corvusoft:asio-dependency:f5c5708:a/dependency/asio GH_TUPLE line has a tag containing something else

This was happening with pre-11 versions. In 11 this shouldn't happen.

There should be no funny business with VBox. To save time, I recommend to install 11.1 image from here: https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.1/

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 5, 2017

Ok attempting now.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 14, 2017

Sorry for the delay. I've managed to install 11.1 and I have received a single test failure.

The following tests FAILED:
               73 - service_connection_timeout_acceptance_test_suite (OTHER_FAULT)
@callum-kirby

This comment has been minimized.

Copy link
Contributor

@callum-kirby callum-kirby commented Aug 14, 2017

@ben-crowhurst I resolved issue as requested. No other issues found.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 28, 2017

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Aug 28, 2017

In 9a5c69e, I still get 27% tests passed, 83 tests failed out of 113.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 28, 2017

Would it be possible to create an ISO image of your environment and make available for download?

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Aug 28, 2017

That's what I was also thinking.
Is VirtualBox VM image okay?

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 28, 2017

That would be perfect! Lets put this issue to bed.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Aug 29, 2017

I installed a fresh copy of FreeBSD 11 into the VM, and tests are failing.

VM image (64-bit): http://doctorlan.com/vm-images/FreeBSD-11-restbed.ova
root password: bsd
How to reproduce:

  1. login as root
  2. cd /usr/ports/www/restbed
  3. make test
  4. observe failures

You can install any other packages with pkg.
For example, in order to install vim, run pkg install vim.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Aug 30, 2017

Thanks I'll test this shortly.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Sep 1, 2017

I've managed to replicate failing test cases within the supplied OVA. It appears any network related tests fail.

Have you applied any specific environmental configuration to this device?

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Sep 1, 2017

Error from HTTP client Failed to locate HTTP endpoint: Connection refused.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Sep 1, 2017

For some reason on this device localhost is not being translated. using IPv6 address ::1 works well; please review your system configuration.

@ben-crowhurst ben-crowhurst self-assigned this Sep 1, 2017
@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Sep 1, 2017

For some reason on this device localhost is not being translated.

Translation is only guaranteed on linux. On BSD it is driven by a special sysctl variable that is off by default.
Please do not rely on or assume IPv6/IPv4 address translation.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Sep 1, 2017

The sysctl variable is net.inet6.ip6.v6only. It is 1 by default, and should be 0 for translation to take place. It should be assumed to be =1 in general.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Sep 1, 2017

All tests indeed pass with net.inet6.ip6.v6only=0.

You need to provide a compile-time variable (ex. IPV4_IPV6_MAPPING_SUPPORTED), so that non-linux systems can run restbed too.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Sep 2, 2017

This seems like a religious debate in the making.

Let me look into the matter further and determine the responsibility user, restbed, or asio dependency.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Sep 2, 2017

Either way I'll look into having a test to specifically probe this scenario.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Sep 2, 2017

You were already aware of the cause?

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Sep 2, 2017

I encountered the problem before (the link above), but didn't connect this case to it until you pointed out that IPv4/IPv6 mapping is failing.

BSDs tend to do things "the right way". I also happen to believe that such mapping is unreasonable, and shouldn't be in place by default, because the only reason is 'cuz Linux does this'.

IPv4 and IPv6 have very little in common. No need to map them into each other. You shouldn't write the code under the assumption that such mapping is always present.

@yurivict

This comment has been minimized.

Copy link
Author

@yurivict yurivict commented Jan 13, 2018

How to proceed with the FreeBSD port?
Tests break because of the flaw in the upstream design: it assumes IPv4/IPv6 mapping when it doesn't exist.
You should not assume such mapping.

@ben-crowhurst

This comment has been minimized.

Copy link
Member

@ben-crowhurst ben-crowhurst commented Jan 15, 2018

@yurivict thanks for your patients. We are currently very busy and will look to rectify this issue when time permits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.