Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build environment updates #145

Closed
wants to merge 8 commits into from
Closed

Build environment updates #145

wants to merge 8 commits into from

Conversation

mbland
Copy link
Contributor

@mbland mbland commented Jun 26, 2014

This pull contains commits to fix the OS X build and allow GitMake test to pass.


# Compatible with -j...
$str =~ s/^(\s+@?)/$1cd \$(TEST_D) && /;
return $str;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect removing this will break FreeBSD.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite the opposite; ./GitMake test was broken for me everywhere until I removed these lines. I just did a full 'make test' and 'GitMake test' on FreeBSD 9.1 to make certain that it works.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On 9 July 2014 21:44, Mike Bland notifications@github.com wrote:

In util/pl/unix.pl:

@@ -226,12 +228,6 @@ sub fixdeps
sub fixrules
{

my ($str) = @_;

  • Compatible with -j...

  • $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /;
  • return $str;

Quite the opposite; ./GitMake test was broken for me everywhere until I
removed these lines. I just did a full 'make test' and 'GitMake test' on
FreeBSD 9.1 to make certain that it works.

Did you do GitMake -j 20 test?


Reply to this email directly or view it on GitHub
https://github.com/openssl/openssl/pull/145/files#r14734751.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a 'GitMake -j 12' followed by a 'GitMake test' before writing.

Just now, I checked out a clean branch based on the upstream master. First, I did a 'GitMake -j 20' followed by 'GitMake test'. The 'GitMake test' step broke.

Then I ran 'GitMake -j 20' test. It also broke.

Back in my branch containing these changes, 'GitMake -j 20' passed. This is all on FreeBSD 9.1.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well. I pulled these changes and did GitMake -j 1 -P test (which works for me) and I get:

test SSL protocol
cd out.mbland-build && /usr/home/ben/git-work/openssl/out.mbland-build/shlib_wrap.sh ./ssltest -test_cipherlist
cd: out.mbland-build: No such file or directory

cd: out.mbland-build: No such file or directory

cd: out.mbland-build: No such file or directory

cd: out.mbland-build: No such file or directory

test_ssl test done
Remaking `test_req'
Results of making test_req:

test_req test started

testing req conversions
p -> d
*** [test_req] Error code 1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On 10 July 2014 19:38, Mike Bland notifications@github.com wrote:

In util/pl/unix.pl:

@@ -226,12 +228,6 @@ sub fixdeps
sub fixrules
{

my ($str) = @_;

  • Compatible with -j...

  • $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /;
  • return $str;

I said it worked because originally I wasn't using '-j', so it was working
for me. When I used '-j 20', I didn't see the "cd: out.build: No such file
or directory" errors and the shell return code was 0.

That's what's puzzling me - I get you could miss the text-only errors, but
I got an actual make-stopping error, and I'm wondering why you didn't?

Perhaps try that on your end to verify; with the (now obsolete) change, it
works, no errors. But then, it defeats the purpose of using '-j' to
parallelize the build by breaking that case.

Yeah, I know -P doesn't change behavior; it's just interesting that it's
not present in all version of bsdmake anymore, not even between versions of
FreeBSD. :-)


Reply to this email directly or view it on GitHub
https://github.com/openssl/openssl/pull/145/files#r14784424.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Thu, Jul 10, 2014 at 3:42 PM, Ben Laurie notifications@github.com wrote:

In util/pl/unix.pl:

@@ -226,12 +228,6 @@ sub fixdeps
sub fixrules
{

my ($str) = @_;

  • Compatible with -j...

  • $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /;
  • return $str;

On 10 July 2014 19:38, Mike Bland notifications@github.com wrote: In util/pl/unix.pl: > @@ -226,12 +228,6 @@ sub fixdeps > sub fixrules > { > my ($str) = @_; > - > - # Compatible with -j... > - $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /; > - return $str; I said it worked because originally I wasn't using '-j', so it was working for me. When I used '-j 20', I didn't see the "cd: out.build: No such file or directory" errors and the shell return code was 0.

That's what's puzzling me - I get you could miss the text-only errors, but I got an actual make-stopping error, and I'm wondering why you didn't?

There is other GitMake weirdness going on that I'm investigating,
which I was going to raise in a separate thread later, but since it
seems to be germane here...

While beginning my investigation into your earlier report this
morning, I noticed that in a brand new working directory identical to
the latest OpenSSL master branch, if I do a 'GitMake -j 1',
test_verify fails. If I then do a regular make, the subsequent
'GitMake -j 1 test_verify' will pass.

It seems something similar is happening with 'GitMake -j 1 test' in
general. I use git-new-workdir to have a private dir per branch. In my
original FreeBSD 9.1 working dir for this 'build' branch, running
'GitMake -j 1 test' even now (after re-introducing the obsolete
change), I'll see the 'cd' errors, but the make will return status
zero. I have previously done a regular 'make' in this directory.

Using the same, identical branch in a different workdir, without first
running 'make', I see the failures and make returns an error.

I'm spending this afternoon digging in to try to understand what the
difference is between the two directories that accounts for this
behavior. It's not obvious to me by diffing the output of 'find . >
files.txt' for each dir.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Thu, Jul 10, 2014 at 4:09 PM, Mike Bland mbland@acm.org wrote:

On Thu, Jul 10, 2014 at 3:42 PM, Ben Laurie notifications@github.com wrote:

In util/pl/unix.pl:

@@ -226,12 +228,6 @@ sub fixdeps
sub fixrules
{

my ($str) = @_;

  • Compatible with -j...

  • $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /;
  • return $str;

On 10 July 2014 19:38, Mike Bland notifications@github.com wrote: In util/pl/unix.pl: > @@ -226,12 +228,6 @@ sub fixdeps > sub fixrules > { > my ($str) = @_; > - > - # Compatible with -j... > - $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /; > - return $str; I said it worked because originally I wasn't using '-j', so it was working for me. When I used '-j 20', I didn't see the "cd: out.build: No such file or directory" errors and the shell return code was 0.

That's what's puzzling me - I get you could miss the text-only errors, but I got an actual make-stopping error, and I'm wondering why you didn't?

There is other GitMake weirdness going on that I'm investigating,
which I was going to raise in a separate thread later, but since it
seems to be germane here...

While beginning my investigation into your earlier report this
morning, I noticed that in a brand new working directory identical to
the latest OpenSSL master branch, if I do a 'GitMake -j 1',
test_verify fails. If I then do a regular make, the subsequent
'GitMake -j 1 test_verify' will pass.

It seems something similar is happening with 'GitMake -j 1 test' in
general. I use git-new-workdir to have a private dir per branch. In my
original FreeBSD 9.1 working dir for this 'build' branch, running
'GitMake -j 1 test' even now (after re-introducing the obsolete
change), I'll see the 'cd' errors, but the make will return status
zero. I have previously done a regular 'make' in this directory.

Using the same, identical branch in a different workdir, without first
running 'make', I see the failures and make returns an error.

I'm spending this afternoon digging in to try to understand what the
difference is between the two directories that accounts for this
behavior. It's not obvious to me by diffing the output of 'find . >
files.txt' for each dir.

FWIW, here's the broken test_verify output I'm getting:

test_verify test started
The following command should have some OK's and some failures
There are definitly a few expired certificates
/Users/msb/src/openssl-fresh/out.build/shlib_wrap.sh
/Users/msb/src/openssl-fresh/out.build/openssl verify -CApath
/Users/msb/src/openssl-fresh/certs/demo
/Users/msb/src/openssl-fresh/certs/demo/.pem
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
140735297401696:error:0B06E06B:x509 certificate
routines:X509_get_pubkey_parameters:unable to find parameters in
chain:/Users/msb/src/openssl-fresh/crypto/x509/x509_vfy.c:2005:
/Users/msb/src/openssl-fresh/certs/demo/ca-cert.pem: C = AU, ST =
Queensland, O = CryptSoft Pty Ltd, CN = Test CA (1024 bit)
error 20 at 0 depth lookup:unable to get local issuer certificate
/Users/msb/src/openssl-fresh/certs/demo/dsa-ca.pem: C = AU, ST =
Some-State, O = Internet Widgits Pty Ltd, CN = CA
error 20 at 0 depth lookup:unable to get local issuer certificate
/Users/msb/src/openssl-fresh/certs/demo/dsa-pca.pem: C = AU, ST =
Some-State, O = Internet Widgits Pty Ltd, CN = PCA
error 18 at 0 depth lookup:self signed certificate
C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = PCA
error 10 at 0 depth lookup:certificate has expired
OK
/Users/msb/src/openssl-fresh/certs/demo/pca-cert.pem: C = AU, ST =
Queensland, O = CryptSoft Pty Ltd, CN = Test PCA (1024 bit)
error 18 at 0 depth lookup:self signed certificate
C = AU, ST = Queensland, O = CryptSoft Pty Ltd, CN = Test PCA (1024 bit)
error 10 at 0 depth lookup:certificate has expired
OK
*
* Error code 2
1 error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Thu, Jul 10, 2014 at 4:15 PM, Mike Bland mbland@acm.org wrote:

On Thu, Jul 10, 2014 at 4:09 PM, Mike Bland mbland@acm.org wrote:

On Thu, Jul 10, 2014 at 3:42 PM, Ben Laurie notifications@github.com wrote:

In util/pl/unix.pl:

@@ -226,12 +228,6 @@ sub fixdeps
sub fixrules
{

my ($str) = @_;

  • Compatible with -j...

  • $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /;
  • return $str;

On 10 July 2014 19:38, Mike Bland notifications@github.com wrote: In util/pl/unix.pl: > @@ -226,12 +228,6 @@ sub fixdeps > sub fixrules > { > my ($str) = @_; > - > - # Compatible with -j... > - $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /; > - return $str; I said it worked because originally I wasn't using '-j', so it was working for me. When I used '-j 20', I didn't see the "cd: out.build: No such file or directory" errors and the shell return code was 0.

That's what's puzzling me - I get you could miss the text-only errors, but I got an actual make-stopping error, and I'm wondering why you didn't?

There is other GitMake weirdness going on that I'm investigating,
which I was going to raise in a separate thread later, but since it
seems to be germane here...

While beginning my investigation into your earlier report this
morning, I noticed that in a brand new working directory identical to
the latest OpenSSL master branch, if I do a 'GitMake -j 1',
test_verify fails. If I then do a regular make, the subsequent
'GitMake -j 1 test_verify' will pass.

It seems something similar is happening with 'GitMake -j 1 test' in
general. I use git-new-workdir to have a private dir per branch. In my
original FreeBSD 9.1 working dir for this 'build' branch, running
'GitMake -j 1 test' even now (after re-introducing the obsolete
change), I'll see the 'cd' errors, but the make will return status
zero. I have previously done a regular 'make' in this directory.

Using the same, identical branch in a different workdir, without first
running 'make', I see the failures and make returns an error.

I'm spending this afternoon digging in to try to understand what the
difference is between the two directories that accounts for this
behavior. It's not obvious to me by diffing the output of 'find . >
files.txt' for each dir.

FWIW, here's the broken test_verify output I'm getting:

test_verify test started
The following command should have some OK's and some failures
There are definitly a few expired certificates
/Users/msb/src/openssl-fresh/out.build/shlib_wrap.sh
/Users/msb/src/openssl-fresh/out.build/openssl verify -CApath
/Users/msb/src/openssl-fresh/certs/demo
/Users/msb/src/openssl-fresh/certs/demo/.pem
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
140735297401696:error:0B06E06B:x509 certificate
routines:X509_get_pubkey_parameters:unable to find parameters in
chain:/Users/msb/src/openssl-fresh/crypto/x509/x509_vfy.c:2005:
/Users/msb/src/openssl-fresh/certs/demo/ca-cert.pem: C = AU, ST =
Queensland, O = CryptSoft Pty Ltd, CN = Test CA (1024 bit)
error 20 at 0 depth lookup:unable to get local issuer certificate
/Users/msb/src/openssl-fresh/certs/demo/dsa-ca.pem: C = AU, ST =
Some-State, O = Internet Widgits Pty Ltd, CN = CA
error 20 at 0 depth lookup:unable to get local issuer certificate
/Users/msb/src/openssl-fresh/certs/demo/dsa-pca.pem: C = AU, ST =
Some-State, O = Internet Widgits Pty Ltd, CN = PCA
error 18 at 0 depth lookup:self signed certificate
C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = PCA
error 10 at 0 depth lookup:certificate has expired
OK
/Users/msb/src/openssl-fresh/certs/demo/pca-cert.pem: C = AU, ST =
Queensland, O = CryptSoft Pty Ltd, CN = Test PCA (1024 bit)
error 18 at 0 depth lookup:self signed certificate
C = AU, ST = Queensland, O = CryptSoft Pty Ltd, CN = Test PCA (1024 bit)
error 10 at 0 depth lookup:certificate has expired
OK
*
* Error code 2
1 error

Figured out the case of test_verify. 'make rehash' needs to be invoked
to create symlinks in certs/demo, via 'make build_apps'. Hand-jamming
this into a previously-failing GitMake-only working dir causes
test_verify to pass. So it looks like a dependency disconnect I'll
need to patch up somehow...

Still working on finding out why the entire test set doesn't return an
error code, though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Thu, Jul 10, 2014 at 5:19 PM, Mike Bland mbland@acm.org wrote:

On Thu, Jul 10, 2014 at 4:15 PM, Mike Bland mbland@acm.org wrote:

On Thu, Jul 10, 2014 at 4:09 PM, Mike Bland mbland@acm.org wrote:

On Thu, Jul 10, 2014 at 3:42 PM, Ben Laurie notifications@github.com wrote:

In util/pl/unix.pl:

@@ -226,12 +228,6 @@ sub fixdeps
sub fixrules
{

my ($str) = @_;

  • Compatible with -j...

  • $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /;
  • return $str;

On 10 July 2014 19:38, Mike Bland notifications@github.com wrote: In util/pl/unix.pl: > @@ -226,12 +228,6 @@ sub fixdeps > sub fixrules > { > my ($str) = @_; > - > - # Compatible with -j... > - $str =~ s/^(\s+@?)/$1cd $(TEST_D) && /; > - return $str; I said it worked because originally I wasn't using '-j', so it was working for me. When I used '-j 20', I didn't see the "cd: out.build: No such file or directory" errors and the shell return code was 0.

That's what's puzzling me - I get you could miss the text-only errors, but I got an actual make-stopping error, and I'm wondering why you didn't?

There is other GitMake weirdness going on that I'm investigating,
which I was going to raise in a separate thread later, but since it
seems to be germane here...

While beginning my investigation into your earlier report this
morning, I noticed that in a brand new working directory identical to
the latest OpenSSL master branch, if I do a 'GitMake -j 1',
test_verify fails. If I then do a regular make, the subsequent
'GitMake -j 1 test_verify' will pass.

It seems something similar is happening with 'GitMake -j 1 test' in
general. I use git-new-workdir to have a private dir per branch. In my
original FreeBSD 9.1 working dir for this 'build' branch, running
'GitMake -j 1 test' even now (after re-introducing the obsolete
change), I'll see the 'cd' errors, but the make will return status
zero. I have previously done a regular 'make' in this directory.

Using the same, identical branch in a different workdir, without first
running 'make', I see the failures and make returns an error.

I'm spending this afternoon digging in to try to understand what the
difference is between the two directories that accounts for this
behavior. It's not obvious to me by diffing the output of 'find . >
files.txt' for each dir.

FWIW, here's the broken test_verify output I'm getting:

test_verify test started
The following command should have some OK's and some failures
There are definitly a few expired certificates
/Users/msb/src/openssl-fresh/out.build/shlib_wrap.sh
/Users/msb/src/openssl-fresh/out.build/openssl verify -CApath
/Users/msb/src/openssl-fresh/certs/demo
/Users/msb/src/openssl-fresh/certs/demo/.pem
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
140735297401696:error:0B06E06B:x509 certificate
routines:X509_get_pubkey_parameters:unable to find parameters in
chain:/Users/msb/src/openssl-fresh/crypto/x509/x509_vfy.c:2005:
/Users/msb/src/openssl-fresh/certs/demo/ca-cert.pem: C = AU, ST =
Queensland, O = CryptSoft Pty Ltd, CN = Test CA (1024 bit)
error 20 at 0 depth lookup:unable to get local issuer certificate
/Users/msb/src/openssl-fresh/certs/demo/dsa-ca.pem: C = AU, ST =
Some-State, O = Internet Widgits Pty Ltd, CN = CA
error 20 at 0 depth lookup:unable to get local issuer certificate
/Users/msb/src/openssl-fresh/certs/demo/dsa-pca.pem: C = AU, ST =
Some-State, O = Internet Widgits Pty Ltd, CN = PCA
error 18 at 0 depth lookup:self signed certificate
C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = PCA
error 10 at 0 depth lookup:certificate has expired
OK
/Users/msb/src/openssl-fresh/certs/demo/pca-cert.pem: C = AU, ST =
Queensland, O = CryptSoft Pty Ltd, CN = Test PCA (1024 bit)
error 18 at 0 depth lookup:self signed certificate
C = AU, ST = Queensland, O = CryptSoft Pty Ltd, CN = Test PCA (1024 bit)
error 10 at 0 depth lookup:certificate has expired
OK
*
* Error code 2
1 error

Figured out the case of test_verify. 'make rehash' needs to be invoked
to create symlinks in certs/demo, via 'make build_apps'. Hand-jamming
this into a previously-failing GitMake-only working dir causes
test_verify to pass. So it looks like a dependency disconnect I'll
need to patch up somehow...

Still working on finding out why the entire test set doesn't return an
error code, though.

Got it. In a clean working dir, with this change, test{req,key}.pem
aren't being generated into out.build. If you do 'GitMake -j 1 -P
test_req' specifically, you'll see the prerequisite _fake1 target
(which generates the pem files) produce nothing yet still indicate a
successful result, which then causes test_req to break.

In my original working dir, the files were there from a previous build
before this change, so the test_req target continued to "pass" in
'GitMake -j' mode. Removing them caused the exact same failure and
output you described earlier.

cscope.out is generated by cscope as described in:
http://wiki.openssl.org/index.php/Testing_and_Development_Tools_and_Tips

.d files are compiler-generated Makefile dependency files (e.g. using
'gcc -MMD -MP foo.c').

Signed-off-by: Mike Bland <mbland@acm.org>
This fixes the errors when trying to assemble .s files using GitMake on OS X.

Signed-off-by: Mike Bland <mbland@acm.org>
These are based on debug-ben-debug-64-clang and is intended to produce
consistent settings for folks involved in the unit testing effort detailed at:

http://wiki.openssl.org/index.php/Unit_Testing

-fsanitize has been removed from the set of clang flags for now. Apparently
clang 3.1, which ships with FreeBSD 9.1, completely ignores -fsanitize. Clang
3.3, which ships with FreeBSD 9.2, compiles with it, but fails to link due to
the absence of libasan:

http://lists.freebsd.org/pipermail/freebsd-hackers/2013-December/043995.html
https://www.mail-archive.com/cfe-commits@cs.uiuc.edu/msg92260.html
http://reviews.llvm.org/D2644

We need -Wno-error=unused-const-variable because of this error:
.../crypto/ec/ec_lib.c:74:19: error: unused variable 'EC_version' [-Werror,-Wunused-const-variable]
static const char EC_version[] = "EC" OPENSSL_VERSION_PTEXT;

Signed-off-by: Mike Bland <mbland@acm.org>
Before this change, variables for which a '=' appeared in the assignment would
be parsed as the entire string up until the final '='. For example:

  BUILD_CMD=shlib_target=; if [ -n "$(SHARED_LIBS)" ]; then \

would result in the variable name "BUILD_CMD=shlib_target". This doesn't
appear to harm the current generation of MINFO, but creates problems for other
Makefile-related work I'm attempting.

Signed-off-by: Mike Bland <mbland@acm.org>
These correspond to targets of the same name in test/Makefile that clash when
using the single-makefile build method using GitConfigure and GitMake.

Signed-off-by: Mike Bland <mbland@acm.org>
This is the only Makefile without SRC defined. This change enables a
standard Makefile include directive to cover crypto/jpake/*.d files.

This was automatically applied by AddSrcVarIfNeeded() in:
https://code.google.com/p/mike-bland/source/browse/openssl/update_makefiles.py

Signed-off-by: Mike Bland <mbland@acm.org>
Signed-off-by: Mike Bland <mbland@acm.org>
This should have been $(CC) all along; removing $(BUILDENV) from the
make invocation causes ${{CC}} to be empty, triggering a build failure as
LDCMD (in Makefile.shared) receives no value.

Signed-off-by: Mike Bland <mbland@acm.org>
@levitte
Copy link
Member

levitte commented Mar 9, 2015

It appears all of the changes are in master, so I'm closing this.

@levitte levitte closed this Mar 9, 2015
mamckee pushed a commit to mamckee/openssl that referenced this pull request Dec 28, 2022
* Change to build with CMake

* Fix problems during shared library test

* Try to find and install libraries correctly

* Don't delete OQS symlinks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants