cleanup #9
Merged
cleanup #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yairchn
force-pushed
the
yc/cleanup
branch
3 times, most recently
from
14c901c
to
41fafaf
Compare
clean EDMF_Environment.jl clean all if S commands cleanup unused closures back to 7388e90 unused closures clean unused function remove diagnostic updratfs remove diagnostic updratfs bool remove bulksteady clean entr_strcat clean entr_strcat clean pres struct some extra cleanups remove buoyant_stract remove buoyant_stract
|
bors r+ |
|
Build succeeded: |
bors bot
added a commit
that referenced
this pull request
Mar 6, 2022
876: Make artifacts download lazily r=charleskawczynski a=charleskawczynski A long time ago, we decided to add a flag in ArtifactWrappers, `local_run`, which allows users to make a temporary download folder to avoid race conditions (downloading Artifacts is not safe to race conditions). The problem is that this basically completely side-steps the utility of lazily downloading the artifacts on remote machines. So, right now, users leverage the utility of Artifacts because the `local_run` flag we have set to `isempty(get(ENV, "CI", ""))`, which is true on buildkite and GHA CI, but all of CI infra is downloading these artifacts on every build. This PR changes this by doing a proper lazy download during the initial `steps` in the buildkite pipeline (which happens in serial, but avoids the race condition). The advantage of this is that the very first time we do this, we'll pay the cost, but subsequent PRs should now be doing a lazy download. We should probably just remove this flag in ArtifactWrappers, since it kind of defeats the purpose of Artifacts to begin with, the logic of always downloading can be done by users. With this PR, we should basically no longer run into these failures: ``` ERROR: LoadError: HTTP/2 302 (Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds) while requesting https://caltech.box.com/shared/static/7upt639siyc2umon8gs6qsjiqavof5cq.nc -- | Stacktrace: | [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy) | @ Downloads /central/software/julia/1.7.0/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:369 | [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, I ``` Also, we should probably apply this fix to other repos that have used this pattern. Co-authored-by: Charles Kawczynski <kawczynski.charles@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
this PR is a running PR to clean the code and validate in buildkite that is keeps passing all tests