New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing Azure Pipelines #15280

Merged
merged 3 commits into from Sep 23, 2018

Conversation

Projects
None yet
7 participants
@azure-pipelines
Contributor

azure-pipelines bot commented Sep 23, 2018

NO ENTRY

@sympy-bot

This comment has been minimized.

sympy-bot commented Sep 23, 2018

Hi, I am the SymPy bot (v133). I'm here to help you write a release notes entry. Please read the guide on how to write release notes.

  • No release notes entry will be added for this pull request.

Note: This comment will be updated with the latest check if you edit the pull request. You need to reload the page to see it.

Click here to see the pull request description that was parsed.

<!-- BEGIN RELEASE NOTES -->
NO ENTRY
<!-- END RELEASE NOTES -->

asmeurer added some commits Sep 23, 2018

@smichr smichr merged commit 5e1260b into master Sep 23, 2018

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
sympy-bot/release-notes The release notes look OK
Details

@asmeurer asmeurer deleted the azure-pipelines branch Sep 23, 2018

@sidhantnagpal

This comment has been minimized.

Member

sidhantnagpal commented Sep 25, 2018

This could possibly be a required check for the Pull Requests. Reference to #15286 for tests failing on Azure but passing on Travis.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Sep 25, 2018

Well I wasn't actually finished here. The idea is to move a good chunk of the tests to Azure, depending on how well it works, we can move everything.

Why did a test fail on Azure and pass on Travis? Was it a random failure?

@sidhantnagpal

This comment has been minimized.

Member

sidhantnagpal commented Sep 25, 2018

The job failed due to the presence of unicode characters in the docstring. The log is accessible for previous builds and it reads:

2018-09-24T17:35:37.2415761Z ==============================================================================
2018-09-24T17:35:37.2428174Z Task         : Command Line
2018-09-24T17:35:37.2441469Z Description  : Run a command line script using cmd.exe on Windows and bash on macOS and Linux.
2018-09-24T17:35:37.2455406Z Version      : 2.136.0
2018-09-24T17:35:37.2467766Z Author       : Microsoft Corporation
2018-09-24T17:35:37.2480743Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2018-09-24T17:35:37.2494537Z ==============================================================================
2018-09-24T17:35:37.5018528Z Generating script.
2018-09-24T17:35:37.5054247Z Script contents:
2018-09-24T17:35:37.5066977Z ./bin/doctest
2018-09-24T17:35:37.5193102Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/fd072d0b-772c-4c70-9d2b-1127c5844bb1.sh
2018-09-24T17:35:39.3808677Z Traceback (most recent call last):
2018-09-24T17:35:39.3831322Z   File "./bin/doctest", line 73, in <module>
2018-09-24T17:35:39.3846936Z     import sympy
2018-09-24T17:35:39.3859576Z   File "/home/vsts/work/1/s/sympy/__init__.py", line 70, in <module>
2018-09-24T17:35:39.3872601Z     from .geometry import *
2018-09-24T17:35:39.3884871Z   File "/home/vsts/work/1/s/sympy/geometry/__init__.py", line 17, in <module>
2018-09-24T17:35:39.3898630Z     from sympy.geometry.ellipse import Ellipse, Circle
2018-09-24T17:35:39.3912209Z   File "/home/vsts/work/1/s/sympy/geometry/ellipse.py", line 452
2018-09-24T17:35:39.3926752Z SyntaxError: Non-ASCII character '\xe2' in file /home/vsts/work/1/s/sympy/geometry/ellipse.py on line 453, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

I am not sure why it didn't show up on Travis.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Sep 25, 2018

Interesting. In Python 2, if you use Unicode characters anywhere in the file you have to put a # -*- coding: utf-8 -*- line at the top. I'm not clear why it didn't fail on Travis. Maybe something is configured somehow so there is a default encoding there.

@moorepants

This comment has been minimized.

Member

moorepants commented Sep 25, 2018

I don't see anything on line 452 in the PR that looks suspect.

One thing that I discovered recently was that Python 3 will default to the system's encoding if you open files without using the new open(encoding=) kwarg. We were caught by this trying to run some code on FreeBSD that ran fine on Ubuntu. Just mentioning since MS Azure may have a different default encoding.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Sep 25, 2018

It was fixed in 17ef36b. It's hard to tell, but the character that was there, , is not the same as the ASCII -.

>>> import unicodedata
>>> unicodedata.name('')
'MINUS SIGN'
>>> unicodedata.name('-')
'HYPHEN-MINUS'
>>> ord('')
8722
>>> ord('-')
45
@asmeurer

This comment has been minimized.

Member

asmeurer commented Sep 25, 2018

Also see https://dev.azure.com/asmeurer/SymPy/_build/results?buildId=19&view=logs. It failed in Python 2, not Python 3. What you mentioned can be an issue, @moorepants, but that isn't what happened here (there is no file opening).

@vtbassmatt

This comment has been minimized.

vtbassmatt commented Oct 1, 2018

Hey folks. I'm a PM on Azure Pipelines. It looks like you've tracked this down to a failing testcase, but if there's anything I can do to help, please shout!

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 1, 2018

@vtbassmatt yes there are no issues with Azure for this issue (if anything, there is something going on with Travis because the failure didn't show up there).

Right now we are trying to translate our Travis build matrix to pipelines. #15297. I'm still trying to find the relevant part of the docs to figure out how to set up the matrix.

I also have a question that maybe you know the answer to. Is it possible to make it so that the azure pipelines link at the bottom of the pull request goes directly to the Azure pipelines website? Right now it goes to the GitHub checks tab, which is useless, because it doesn't contain the logs for the build.

@vtbassmatt

This comment has been minimized.

vtbassmatt commented Oct 2, 2018

Your Azure Pipelines file correctly tests Pythons 2.7 and 3.5 - 3.7, but I see that you also do pypy on the old system. Is that what you want to add?

I looked at your Travis CI file, and I see all the SPLIT stuff. I'm honestly not clear what it's trying to do, so I can't give any better advice there.

I'm not sure about the link situation. On all the repos I checked, the Azure Pipelines link goes directly to the status page. And I don't see a check configured for any of the open PRs in this repo, so I can't take a look there. Do you have an example of a PR that's still open and exhibiting this?

Thanks!

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 2, 2018

The SPLIT thing is something we use in our test runner to keep the test time under Travis's limit. We might end up removing that for Azure. Anyway it isn't relevant to my question. You can see at https://travis-ci.org/sympy/sympy we test everything under 2.7-3.6 except the later (TEST_ASCII=true, TEST_OPT_DEPENDENCY=..., and also TEST_SLOW) which are only under 2.7 and 3.6, because they take a while and aren't worth testing on the intermediate versions. So I'm unsure how to do that in Azure, because the YAML file has 4 Python versions in the matrix. I basically want to have a build matrix where some combinations are excluded.

I'm not sure about the link situation. On all the repos I checked, the Azure Pipelines link goes directly to the status page. And I don't see a check configured for any of the open PRs in this repo, so I can't take a look there. Do you have an example of a PR that's still open and exhibiting this?

Yeah, that's because I accidentally enabled it under my own username on Azure, instead of under a SymPy organization. So I deleted it and readded it, but I guess it didn't work. Right now my current work is at #15297. You can see there if you click the azure pipelines "details" link it takes you to https://github.com/sympy/sympy/pull/15297/checks?check_run_id=19799837, which doesn't have any useful information. I would rather it just went directly to https://dev.azure.com/asmeurer/6bd85cc2-5c5a-4f85-a49f-3744cf16d5be/_build/results?buildId=57.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 2, 2018

I guess I needed to trigger a build first. Once I did that it started building from the PR again.

I think I figured out the matrix thing (I just need to have separate jobs). I'll let you know if I have any other problems.

Another question: I noticed there is a "teams" concept in Azure. Does that automatically inherit from GitHub permissions like Travis does, or do I need to set it up manually? On Travis, anyone with push access has the ability to restart a build, which I'd like here too (though if Azure works as well as I'm hoping it does we shouldn't need too many build restarts).

@vtbassmatt

This comment has been minimized.

vtbassmatt commented Oct 3, 2018

Hey @asmeurer - we opted to make matrix generation be explicit. We observed that on most services where you can say "give me all versions from m to n", there are nearly always some exceptions, special configs, etc. for one or more versions. I see what you did with separate jobs - that makes sense to me. As an alternative, you could say something like:

strategy:
  matrix:
    Python27:
      python.version: '2.7'
      TEST_ASCII: true
    Python35:
      python.version: '3.5'
      TEST_ASCII: false
    Python36:
      python.version: '3.6'
      TEST_ASCII: true
    Python37:
      python.version: '3.7'
      TEST_ASCII: false
...
steps:
- script: ./bin/doctest
  condition: eq(TEST_ASCII, 'true')
...

(Written from memory so I didn't explicitly test this, since I think your solution is probably cleaner.)

We don't carry over GitHub permissions. For someone without access to your Azure Pipelines service, the way to re-queue a build is going to be to push a trivial change, or ask someone to click the re-queue button. Definitely a gap in the offering, and something we're looking at fixing.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 3, 2018

I'll take a look at the conditions. My main concern is the duplication of steps between jobs, but I think that might be able to fix it.

Do you have any comments on the checks tab?

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 3, 2018

More questions (please let me know if there's a more appropriate place to ask these than asking you):

  • There doesn't seem to be a way to hotlink a specific build log. Like at https://dev.azure.com/SymPy/SymPy/_build/results?buildId=14&view=logs I want to link to the Sage failure, ideally even the specific line in the log. But the link in the URL bar just goes to the full build with all the jobs.

  • Is there anything like automatic build cancellation from Travis. If you don't know, on Travis you can enable automatic build cancellation and if someone pushes two consecutive commits to the same PR and the build for the earlier one hasn't started yet, it gets cancelled automatically.

@vtbassmatt

This comment has been minimized.

vtbassmatt commented Oct 3, 2018

This is a perfectly fine place to ask questions. If you have longer-form stuff you want to discuss, I'm mattc at xbox.com.

@DavidStaheli any idea why this PR links you to the GitHub Checks tab instead of directly to the Azure Pipelines build status?

@vijayma / @azureDaveOps another request for linking to logs, including specific lines.

We have automatic cancellation for PR builds. It will even cancel running builds, which may be overly aggressive in some cases. We're looking at finer grained control over this behavior.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 3, 2018

OK great. I think for us cancelling a running build is fine, though I can definitely see that being an issue for some other use-cases.

@DavidStaheli

This comment has been minimized.

DavidStaheli commented Oct 3, 2018

About the PR conversation page linking to the Checks tab instead of directly to Azure Pipelines logs, that's GitHub's chosen UX. When you use our GitHub App, we publish build status to GitHub's new Checks API. GitHub then intentionally links to the Checks tab. To get to the Azure Pipelines logs, you have to follow the links on the Checks tab. One good thing is that you can re-run the build from the Checks tab. We currently post test failures there, but we can post more error information there too so that sometimes you can skip going deep in the logs. @kelliejos FYI.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 3, 2018

I don't see very much information posted to the checks tab https://github.com/sympy/sympy/runs/21313012.

I guess I can authenticate via a user instead of the app. I wasn't keen to do that because it requires write access to all my repositories, even from other organizations. But maybe I can do it through a dummy user, if it will fix the problem.

@asmeurer

This comment has been minimized.

Member

asmeurer commented Oct 3, 2018

I just pushed two consecutive commits to #15297 and the build wasn't cancelled (https://dev.azure.com/SymPy/SymPy/_build/results?buildId=40&view=logs should have been cancelled by https://dev.azure.com/SymPy/SymPy/_build/results?buildId=41&view=logs).

@vtbassmatt

This comment has been minimized.

vtbassmatt commented Oct 4, 2018

Hmm, that's surprising to me. We've actually had customers complain that we cancel builds too aggressively. @DavidStaheli again for his thoughts.

@DavidStaheli

This comment has been minimized.

DavidStaheli commented Oct 4, 2018

@asmeurer we're looking into why this build wasn't cancelled right now. Thanks for the heads-up.

@DavidStaheli

This comment has been minimized.

DavidStaheli commented Oct 4, 2018

Hi @asmeurer, I just learned more about what's happening. The behavior you see occurs with our new GitHub App, but not when your pipelines are configured to use OAuth to connect to GitHub. There's a bug in the GitHub App's event handling and a fix is underway. I expect the fix will be available next week.

@asmeurer asmeurer referenced this pull request Oct 12, 2018

Open

Add missing things to Azure pipelines #15385

0 of 5 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment