Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Users/stfrance/add containerfetch timeout envvar #1132

Closed

Conversation

stephenmichaelf
Copy link
Member

This is in reference to #1030

I added a new environment variable that can override the GetFileAsyncTimeout for the ContainerFetchEngine.

@GitHubSriramB please review and see if this is an appropriate solution.

@TingluoHuang
Copy link
Contributor

@stephenmichaelf search for string 'GetFileAsyncTimeoutMinutes', that string will print out on timeout, that int is always 5, i guess you need make some change there to make it print the right timeout. :)

@TingluoHuang
Copy link
Contributor

@GitHubSriramB please take a look.

Copy link
Contributor

@GitHubSriramB GitHubSriramB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@omeshp - can you also take a look at this change?

@@ -236,8 +236,20 @@ private bool Match(ServerBuildArtifact buildArtifact, ArtifactDefinition artifac
DownloadBufferSize = executionContext.Variables.Release_Download_BufferSize ?? ContainerFetchEngineDefaultOptions.DownloadBufferSize
};

int fetchEngineTimeoutSeconds;
if (!int.TryParse(Environment.GetEnvironmentVariable("VSTS_FETCHENGINE_TIMEOUT") ?? string.Empty, out fetchEngineTimeoutSeconds))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use VSTS_HTTP_TIMEOUT here instead of introducing another variable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stephenmichaelf my fault, i forgot v2 agent use VSTS_HTTP_TIMEOUT even for file container upload. I agree with @GitHubSriramB, let's just use VSTS_HTTP_TIMEOUT.
check this: https://github.com/Microsoft/vsts-agent/blob/master/src/Agent.Worker/Build/FileContainerServer.cs#L43

@stephenmichaelf
Copy link
Member Author

@GitHubSriramB I added some changes based on feedback, I am not sure if these are correct so please review.

It seems like there are two competing task timeouts:

  1. timeoutTask which is set from ContainerFetchEngineOptions.GetFileAsyncTimeout
  2. getFileTask which uses a VssConnection which has a timeout set from VSTS_HTTP_TIMEOUT

I think what's happening beofore is that #1 was being set from the hardcoded value of GetFileAsyncTimeout in ContainerFetchEngineDefaultOPtions which was lower than VSTS_HTTP_TIMEOUT so that first timeout was firing earlier.

From my understanding there might be more cleanup that can be done but I think at least if we set GetFileAsyncTimeout to respect the VSTS_HTTP_TIMEOUT environment variable then we should solve the problem.

@@ -236,8 +236,20 @@ private bool Match(ServerBuildArtifact buildArtifact, ArtifactDefinition artifac
DownloadBufferSize = executionContext.Variables.Release_Download_BufferSize ?? ContainerFetchEngineDefaultOptions.DownloadBufferSize
};

int fetchEngineTimeoutSeconds;
if (!int.TryParse(Environment.GetEnvironmentVariable("VSTS_HTTP_TIMEOUT") ?? string.Empty, out fetchEngineTimeoutSeconds))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't see why we should cap the max. timeout. instead it is better to not let the file download to timeout. there is already a timeout for the deployment configured and if download exceeds this timeout, it would be canceled by agent.

@stephenmichaelf - is it fine if you let @omeshp to take this fix forward?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GitHubSriramB Sounds good thanks!

@microsoft microsoft deleted a comment from msftclas Sep 27, 2017
prebansa pushed a commit to prebansa/azure-pipelines-agent that referenced this pull request Aug 6, 2019
…os10.14_week31

macos 1014 week 31 updates
@vtbassmatt vtbassmatt deleted the users/stfrance/add-containerfetch-timeout-envvar branch October 16, 2019 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants