Fix fragility issues in all.sh #2031

gilles-peskine-arm · 2018-09-26T16:45:03Z

Fix some issues that don't break all.sh on a pristine checkout, but make it fragile in a tree where you've been doing development.

Fix make WINDOWS_BUILD=1 clean on non-Windows hosts.
In keep-going mode, don't hard-fail on some auxiliary script or test.
Fix Coverity branches fail Travis tests with check-files.py #1691 fix check-files.py: don't recurse into .git #1713: check-files.py complaining about files in .git or in third-party subtrees.
Fix make apidoc looking for files in unintended subdirectories.
Don't fail on a spurious error if ASLR can't be disabled (not an issue on developer machines, but this would let us remove the don't-enforce-ASLR configuration from some CI jobs).

Backports: #2032 for 2.7, #2033 for 2.1.

Precursor to PSA.

The clean rule was not using the correct names for the compiled executable files.

Add record_status in front of the invocation of several scripts where it was missing.

gilles-peskine-arm · 2018-09-26T16:47:53Z

all.sh run on CI: https://jenkins-internal.mbed.com/job/mbedtls-release/717/ → pass

simonbutcher

LGTM

Patater

LGTM

Should all commits be backported? Which ones depend on #930 being backported first?

gilles-peskine-arm · 2018-09-27T08:05:52Z

@Patater The WINDOWS_BUILD fix is for an issue introduced by #930. The others are unrelated, but “don't hard-fail on some auxiliary script” modifies one line that was introduced in #930.

gilles-peskine-arm · 2018-09-27T08:26:00Z

I added one commit for another related issue that I noticed while doing the backports. I'll start CI runs of all.sh this evening.

gilles-peskine-arm · 2018-09-27T11:06:22Z

I made another commit that will allow us to run the CI with ASLR enforced.

Generate the documentation from include and doxygen/input only. Don't get snared by files containing Doxygen comments that lie in other directories such as tests, yotta, crypto/include, ... The only difference this makes in a fresh checkout is that the documentation no longer lists target_config.h. This file is from yotta, does not contain any Doxygen comment, and its inclusion in the rendered documentation was clearly an oversight.

Add if_build_succeeded in front of the invocation of some test runs where it was missing.

test -s can't fail if the subsequent grep succeeds.

We don't need to disable ASLR, so don't try. If gdb tries but fails, the test runs normally, but all.sh then trips up because it sees `warning: Error disabling address space randomization: Operation not permitted` and interprets it as an error that indicates a test failure.

gilles-peskine-arm · 2018-09-28T15:44:04Z

@Patater @sbutcher-arm I rewrote the commit that modifies check-files.py to exclude other directories, including a fix for #1691. The original history is in https://github.com/gilles-peskine-arm/mbedtls/tree/all.sh-fragility-201809-1. Please re-review.

gilles-peskine-arm · 2018-09-28T16:59:17Z

New run of all.sh at https://jenkins-internal.mbed.com/job/mbedtls-release/727/ → pass

Patater · 2018-10-02T11:02:34Z

tests/scripts/check-files.py

+        self.excluded_paths = list(map(os.path.normpath, [
+            'cov-int',
+            'examples',
+            'yotta/module'


The commit message said yotta/modules. Should this be module or modules? Update here or the commit message to be correct.

It's module. I amended the last commit to fix the commit message.

The PR to remove yotta support will be merged before this one. I don't think you need that line.

And as an aside - there's no need to block it because check-files.py passes that directory in our repo anyway.

Exclude ".git" directories anywhere. This avoids spurious errors in git checkouts that contain branch names that look like a file check-files.py would check. Fix Mbed-TLS#1713 Exclude "mbed-os" anywhere and "examples" from the root. Switch to the new mechanism to exclude "yotta/module". These are directories where we store third-party files that do not need to match our preferences. Exclude "cov-int" from the root. Fix Mbed-TLS#1691

mpg · 2018-10-16T10:38:24Z

I took the liberty of editing the PR description by adding "fix" in front of an issue number so that github picks it up as fixed by this PR.

simonbutcher

I think you should remove yotta support, but it's not a blocker.

simonbutcher · 2018-10-16T11:21:09Z

tests/scripts/check-files.py

+        self.excluded_paths = list(map(os.path.normpath, [
+            'cov-int',
+            'examples',
+            'yotta/module'


The PR to remove yotta support will be merged before this one. I don't think you need that line.

Complements "Remove Yotta support from the docs, tests and build scripts".

gilles-peskine-arm · 2018-10-16T12:13:03Z

@sbutcher-arm Thanks for reviewing. I removed the yotta line from check-files.py.

dgreen-arm

LGTM

mpg · 2018-10-24T10:27:45Z

retest

simonbutcher · 2018-10-24T12:39:11Z

retest

simonbutcher · 2018-10-25T13:42:23Z

Jenkins failed on the BSD timing test, which is a known issue and as good as a pass. Therefore marking as 'ready for merge'.

gilles-peskine-arm added 2 commits September 26, 2018 16:50

Fix "make WINDOWS_BUILD=1 clean" on non-Windows hosts

2561a50

The clean rule was not using the correct names for the compiled executable files.

In keep-going mode, don't hard-fail on some auxiliary script

487cdf5

Add record_status in front of the invocation of several scripts where it was missing.

gilles-peskine-arm added bug mbed TLS team needs-review Every commit must be reviewed by at least two team members, needs-backports Backports are missing or are pending review and approval. component-platform Portability layer and build scripts labels Sep 26, 2018

simonbutcher previously approved these changes Sep 26, 2018

View reviewed changes

Patater previously approved these changes Sep 26, 2018

View reviewed changes

simonbutcher removed the needs-review Every commit must be reviewed by at least two team members, label Sep 26, 2018

gilles-peskine-arm dismissed stale reviews from Patater and simonbutcher via f12f9ca September 27, 2018 08:15

This was referenced Sep 27, 2018

Backport 2.7: Fix fragility issues in all.sh #2032

Merged

Backport 2.1: Fix fragility issues in all.sh #2033

Merged

gilles-peskine-arm added the needs-review Every commit must be reviewed by at least two team members, label Sep 27, 2018

Patater previously approved these changes Sep 27, 2018

View reviewed changes

simonbutcher previously approved these changes Sep 28, 2018

View reviewed changes

simonbutcher added approved Design and code approved - may be waiting for CI or backports and removed needs-backports Backports are missing or are pending review and approval. needs-review Every commit must be reviewed by at least two team members, labels Sep 28, 2018

gilles-peskine-arm added 4 commits September 28, 2018 14:31

In keep-going mode, don't hard-fail on some tests

bd90a8c

Add if_build_succeeded in front of the invocation of some test runs where it was missing.

Remove redundant check in all.sh

5c39d7a

test -s can't fail if the subsequent grep succeeds.

gilles-peskine-arm dismissed stale reviews from simonbutcher and Patater via d4e40f8 September 28, 2018 15:34

gilles-peskine-arm force-pushed the all.sh-fragility-201809 branch from 25e6c1d to d4e40f8 Compare September 28, 2018 15:34

gilles-peskine-arm added needs-review Every commit must be reviewed by at least two team members, needs-backports Backports are missing or are pending review and approval. and removed approved Design and code approved - may be waiting for CI or backports labels Sep 28, 2018

gilles-peskine-arm mentioned this pull request Oct 2, 2018

2.1 only: add tests/scripts/doxygen.sh and fix make apidoc #2011

Merged

Patater suggested changes Oct 2, 2018

View reviewed changes

gilles-peskine-arm force-pushed the all.sh-fragility-201809 branch from d4e40f8 to 95c5575 Compare October 2, 2018 11:14

Patater previously approved these changes Oct 2, 2018

View reviewed changes

simonbutcher previously approved these changes Oct 16, 2018

View reviewed changes

Remove yotta support from check-files.py

96f3b4e

Complements "Remove Yotta support from the docs, tests and build scripts".

gilles-peskine-arm dismissed stale reviews from simonbutcher and Patater via 96f3b4e October 16, 2018 12:12

dgreen-arm approved these changes Oct 16, 2018

View reviewed changes

simonbutcher approved these changes Oct 22, 2018

View reviewed changes

simonbutcher removed the needs-review Every commit must be reviewed by at least two team members, label Oct 22, 2018

Patater approved these changes Oct 23, 2018

View reviewed changes

simonbutcher added approved Design and code approved - may be waiting for CI or backports and removed needs-backports Backports are missing or are pending review and approval. labels Oct 25, 2018

simonbutcher merged commit 96f3b4e into Mbed-TLS:development Oct 28, 2018

yanesca mentioned this pull request Aug 12, 2019

check-files.py should ignore contents of .git directory #1697

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix fragility issues in all.sh #2031

Fix fragility issues in all.sh #2031

gilles-peskine-arm commented Sep 26, 2018 •

edited by mpg

Loading

gilles-peskine-arm commented Sep 26, 2018 •

edited

Loading

simonbutcher left a comment

Patater left a comment

gilles-peskine-arm commented Sep 27, 2018

gilles-peskine-arm commented Sep 27, 2018

gilles-peskine-arm commented Sep 27, 2018

gilles-peskine-arm commented Sep 28, 2018

gilles-peskine-arm commented Sep 28, 2018 •

edited

Loading

Patater Oct 2, 2018

gilles-peskine-arm Oct 2, 2018

simonbutcher Oct 16, 2018

simonbutcher Oct 16, 2018

mpg commented Oct 16, 2018

simonbutcher left a comment

simonbutcher Oct 16, 2018

gilles-peskine-arm commented Oct 16, 2018

dgreen-arm left a comment

mpg commented Oct 24, 2018

simonbutcher commented Oct 24, 2018

simonbutcher commented Oct 25, 2018

Fix fragility issues in all.sh #2031

Fix fragility issues in all.sh #2031

Conversation

gilles-peskine-arm commented Sep 26, 2018 • edited by mpg Loading

gilles-peskine-arm commented Sep 26, 2018 • edited Loading

simonbutcher left a comment

Choose a reason for hiding this comment

Patater left a comment

Choose a reason for hiding this comment

gilles-peskine-arm commented Sep 27, 2018

gilles-peskine-arm commented Sep 27, 2018

gilles-peskine-arm commented Sep 27, 2018

gilles-peskine-arm commented Sep 28, 2018

gilles-peskine-arm commented Sep 28, 2018 • edited Loading

Patater Oct 2, 2018

Choose a reason for hiding this comment

gilles-peskine-arm Oct 2, 2018

Choose a reason for hiding this comment

simonbutcher Oct 16, 2018

Choose a reason for hiding this comment

simonbutcher Oct 16, 2018

Choose a reason for hiding this comment

mpg commented Oct 16, 2018

simonbutcher left a comment

Choose a reason for hiding this comment

simonbutcher Oct 16, 2018

Choose a reason for hiding this comment

gilles-peskine-arm commented Oct 16, 2018

dgreen-arm left a comment

Choose a reason for hiding this comment

mpg commented Oct 24, 2018

simonbutcher commented Oct 24, 2018

simonbutcher commented Oct 25, 2018

gilles-peskine-arm commented Sep 26, 2018 •

edited by mpg

Loading

gilles-peskine-arm commented Sep 26, 2018 •

edited

Loading

gilles-peskine-arm commented Sep 28, 2018 •

edited

Loading