Skip to content

Conversation

@mkitti
Copy link
Contributor

@mkitti mkitti commented Aug 17, 2021

This will update the Julia feedstock to the current stable version. The initial new commit is from the autotick bot.

The pull request is in draft form since additional commits are expected to take advantage of 1.6.2 build features.

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@mkitti
Copy link
Contributor Author

mkitti commented Aug 18, 2021

Downloads.jl, a part of the Julia standard library encountered three errors when we used the same configuration as Julia 1.6.1.

Details of Downloads.jl errors.
2021-08-17T04:14:54.4131231Z Error in testset Downloads:
2021-08-17T04:14:54.4131759Z Test Failed at $PREFIX/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:387
2021-08-17T04:14:54.4132273Z   Expression: resp isa Response
2021-08-17T04:14:54.4133823Z    Evaluated: Downloads.RequestError("https://wrong.host.badssl.com", 60, "SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'", Downloads.Response("https", "https://wrong.host.badssl.com/", 0, "", Pair{String, String}[])) isa Downloads.Response
2021-08-17T04:14:54.4134974Z Error in testset Downloads:
2021-08-17T04:14:54.4135537Z Error During Test at $PREFIX/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:388
2021-08-17T04:14:54.4136057Z   Test threw exception
2021-08-17T04:14:54.4136472Z   Expression: resp.status == 200
2021-08-17T04:14:54.4136927Z   type RequestError has no field status
2021-08-17T04:14:54.4137707Z   Stacktrace:
2021-08-17T04:14:54.4138190Z    [1] getproperty(x::Downloads.RequestError, f::Symbol)
2021-08-17T04:14:54.4138690Z      @ Base ./Base.jl:33
2021-08-17T04:14:54.4139091Z    [2] macro expansion
2021-08-17T04:14:54.4139962Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:388 [inlined]
2021-08-17T04:14:54.4140854Z    [3] macro expansion
2021-08-17T04:14:54.4141402Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4141976Z    [4] macro expansion
2021-08-17T04:14:54.4142817Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:377 [inlined]
2021-08-17T04:14:54.4143703Z    [5] macro expansion
2021-08-17T04:14:54.4144266Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4144821Z    [6] macro expansion
2021-08-17T04:14:54.4145677Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:367 [inlined]
2021-08-17T04:14:54.4146719Z    [7] macro expansion
2021-08-17T04:14:54.4147270Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4148064Z    [8] top-level scope
2021-08-17T04:14:54.4148954Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:4
2021-08-17T04:14:54.4149845Z Error in testset Downloads:
2021-08-17T04:14:54.4150364Z Test Failed at $PREFIX/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:395
2021-08-17T04:14:54.4150873Z   Expression: resp isa Response
2021-08-17T04:14:54.4152373Z    Evaluated: Downloads.RequestError("https://wrong.host.badssl.com", 60, "SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'", Downloads.Response("https", "https://wrong.host.badssl.com/", 0, "", Pair{String, String}[])) isa Downloads.Response
2021-08-17T04:14:54.4153779Z Error in testset Downloads:
2021-08-17T04:14:54.4154297Z Error During Test at $PREFIX/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:396
2021-08-17T04:14:54.4154813Z   Test threw exception
2021-08-17T04:14:54.4155226Z   Expression: resp.status == 200
2021-08-17T04:14:54.4155684Z   type RequestError has no field status
2021-08-17T04:14:54.4156095Z   Stacktrace:
2021-08-17T04:14:54.4156578Z    [1] getproperty(x::Downloads.RequestError, f::Symbol)
2021-08-17T04:14:54.4157079Z      @ Base ./Base.jl:33
2021-08-17T04:14:54.4157478Z    [2] macro expansion
2021-08-17T04:14:54.4158351Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:396 [inlined]
2021-08-17T04:14:54.4159245Z    [3] macro expansion
2021-08-17T04:14:54.4159795Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4160366Z    [4] macro expansion
2021-08-17T04:14:54.4161334Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:377 [inlined]
2021-08-17T04:14:54.4162237Z    [5] macro expansion
2021-08-17T04:14:54.4162804Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4163359Z    [6] macro expansion
2021-08-17T04:14:54.4164221Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:367 [inlined]
2021-08-17T04:14:54.4165099Z    [7] macro expansion
2021-08-17T04:14:54.4165652Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4166411Z    [8] top-level scope
2021-08-17T04:14:54.4167258Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:4
2021-08-17T04:14:54.4168146Z Error in testset Downloads:
2021-08-17T04:14:54.4168779Z Test Failed at $PREFIX/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:404
2021-08-17T04:14:54.4169288Z   Expression: resp isa Response
2021-08-17T04:14:54.4170799Z    Evaluated: Downloads.RequestError("https://wrong.host.badssl.com", 60, "SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'", Downloads.Response("https", "https://wrong.host.badssl.com/", 0, "", Pair{String, String}[])) isa Downloads.Response
2021-08-17T04:14:54.4171917Z Error in testset Downloads:
2021-08-17T04:14:54.4172431Z Error During Test at $PREFIX/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:405
2021-08-17T04:14:54.4172943Z   Test threw exception
2021-08-17T04:14:54.4173354Z   Expression: resp.status == 200
2021-08-17T04:14:54.4173807Z   type RequestError has no field status
2021-08-17T04:14:54.4174219Z   Stacktrace:
2021-08-17T04:14:54.4174696Z    [1] getproperty(x::Downloads.RequestError, f::Symbol)
2021-08-17T04:14:54.4175209Z      @ Base ./Base.jl:33
2021-08-17T04:14:54.4175612Z    [2] macro expansion
2021-08-17T04:14:54.4176474Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:405 [inlined]
2021-08-17T04:14:54.4177352Z    [3] macro expansion
2021-08-17T04:14:54.4177909Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1226 [inlined]
2021-08-17T04:14:54.4178484Z    [4] macro expansion
2021-08-17T04:14:54.4179326Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:402 [inlined]
2021-08-17T04:14:54.4180206Z    [5] macro expansion
2021-08-17T04:14:54.4180775Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4181332Z    [6] macro expansion
2021-08-17T04:14:54.4182193Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:367 [inlined]
2021-08-17T04:14:54.4183266Z    [7] macro expansion
2021-08-17T04:14:54.4183816Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/work/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
2021-08-17T04:14:54.4184574Z    [8] top-level scope
2021-08-17T04:14:54.4185421Z      @ ~/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:4
2021-08-17T04:14:54.4186332Z ERROR: LoadError: Test run finished with errors
2021-08-17T04:14:54.4187247Z in expression starting at /home/conda/feedstock_root/build_artifacts/julia_1629172591757/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/share/julia/test/runtests.jl:84
2021-08-17T04:14:55.6619941Z ERROR: A test has failed. Please submit a bug report (https://github.com/JuliaLang/julia/issues)
2021-08-17T04:14:55.6621526Z including error messages above and the output of versioninfo():
2021-08-17T04:14:55.6622097Z Julia Version 1.6.2
2021-08-17T04:14:55.6623264Z Commit 1b93d53fc4* (2021-07-14 15:36 UTC)
2021-08-17T04:14:55.6623743Z Platform Info:
2021-08-17T04:14:55.6624370Z   OS: Linux (x86_64-conda-linux-gnu)
2021-08-17T04:14:55.6625129Z   CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
2021-08-17T04:14:55.6625595Z   WORD_SIZE: 64
2021-08-17T04:14:55.6626002Z   LIBM: libopenlibm
2021-08-17T04:14:55.6626642Z   LLVM: libLLVM-11.0.1 (ORCJIT, skylake-avx512)

The errors are due to JuliaLang/Downloads.jl@818cd12 which is a backport of JuliaLang/Downloads.jl#114

Specifically, the test failures occur around lines 387, 395, and 405 of Downloads.jl 1.4.1 which tests a mechanism to override host verification. This is due to only modification of only CURLOPT_SSL_VERIFYPEER and not CURLOPT_SSL_VERIFYHOST. Perhaps the different behavior is due to the use of openssl by conda-forge and mbedtls by official Julia.

By setting CURLOPT_SSL_VERIFYPEER to 0, the expected behavior by the test is that HTTPS request will succeed despite a bad certificate and return a valid Response. Instead, a ResponseError occurs: "SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'.

Subsequent commits upstream in Downloads.jl allow for user control of host verification.

The significance of these test failures are not clear since upstream Julia "have yet to find an example where turning host verification off is necessary." Thus it is likely that b02be7b and 8fe4f7e will be reverted and perhaps the Downloads.jl tests will be dropped.

mkitti added 2 commits August 18, 2021 22:44
…26c5529ea1949c2a5f0937a7e1da unverified HTTPS: don't set CURLOPT_SSL_VERIFYHOST=0"

This reverts commit 8fe4f7e.
recipe/build.sh Outdated
export EXTRA_MAKEFLAGS="USE_SYSTEM_LIBGIT2=0"
fi

# using system utf8proc can be enabled in 1.7.0. See https://github.com/JuliaLang/julia/commit/ba653ecb1c81f1465505c2cea38b4f8149dd20b3
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to use utf8proc in 1.6.2 since that patch was backported.

@mkitti
Copy link
Contributor Author

mkitti commented Aug 19, 2021

The reason the macOS math test are failing at line 296 is that the the tests are running on ivybridge:

Platform Info:
  OS: macOS (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, ivybridge)

https://discourse.julialang.org/t/base-runtests-math-failures-on-chips-older-than-haswell/65954

@mkitti mkitti marked this pull request as ready for review August 20, 2021 06:53
@mkitti
Copy link
Contributor Author

mkitti commented Aug 20, 2021

@isuruf This is a decent working build of Julia 1.6.2. I've dropped the Downloads tests since upstream thinks the failures may be incorrect per JuliaLang/Downloads.jl#140. I've added all the tests for Julia proper except for the standard library tests. The one failing test in math is excluded for osx since the osx virtual machine uses an older ivybridge processor that lacks fused multiply add (FMA) instructions.

I able to use conda-forge's libutf8proc in this build since the femtolisp patch was backported to 1.6.2.

Should one or both of us be listed as maintainers in meta.yaml?

@mkitti
Copy link
Contributor Author

mkitti commented Aug 20, 2021

Note to self, set JULIA_CPU_TARGET to generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)

https://github.com/JuliaCI/julia-buildbot/blob/master/master/inventory.py#L128-L130

@zklaus
Copy link

zklaus commented Sep 1, 2021

@mkitti, given your stellar work on the 1.6 packaging, I definitely support your addition to maintainers, but this is not my package. @conda-forge/julia, any objections?

But I have a question: Is there a particular reason why you opened a new PR for this? Normally, you can work on the PR that was opened by the bot and push to the bot's repo.

@mkitti
Copy link
Contributor Author

mkitti commented Sep 2, 2021

Thanks.

But I have a question: Is there a particular reason why you opened a new PR for this? Normally, you can work on the PR that was opened by the bot and push to the bot's repo.

I was not aware I could push to the bot's branch. Perhaps I need to be a maintainer in order to do that?

@zklaus
Copy link

zklaus commented Sep 2, 2021

I was not aware I could push to the bot's branch. Perhaps I need to be a maintainer in order to do that?

Ah yes, you are probably right. All the more reason to add you 😉

@wolfv
Copy link
Member

wolfv commented Sep 2, 2021

Do you need a member from core to merge this? Would be great to have you as a maintainer @mkitti !

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

  • requirements: host: suitesparse=5.4.0 must contain a space between the name and the pin, i.e. suitesparse =5.4.0

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@mkitti
Copy link
Contributor Author

mkitti commented Sep 2, 2021

Per #118 , we may need to pin some of the dependencies to particular versions. The versions used in the official builds are listed here:
https://github.com/JuliaLang/julia/blob/v1.6.2/deps/Versions.make

@mkitti
Copy link
Contributor Author

mkitti commented Sep 2, 2021

Hi @wolfv , thanks for the offer to merge. I just pushed a few commits addressing some lingering issues. @isuruf reviewed 1.6.1 so it may be useful for him to take a look in case I missed anything.

I would be interested in merging some version of #87 as well, but perhaps we should save that for another build.

Otherwise, I think this is good to merge to have Julia 1.6.2 available via conda-forge, pending passing CI checks.

@mkitti
Copy link
Contributor Author

mkitti commented Sep 7, 2021

This should be ready to merge unless there are other blocking issues. After this, I would like work on another build along the lines of #87 .

recipe/build.sh Outdated
export EXTRA_MAKEFLAGS="USE_SYSTEM_LIBGIT2=1"
fi
# Do this only for x86_64, target platform is not osx-arm64
export JULIA_CPU_TARGET="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what this does?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Here is a quote from https://docs.julialang.org/en/v1/devdocs/sysimg/#Specifying-multiple-system-image-targets

This creates a system image with three separate targets; one for a generic x86_64 processor, one with a sandybridge ISA (explicitly excluding xsaveopt) that explicitly clones all functions, and one targeting the haswell ISA, based off of the sandybridge sysimg version, and also excluding rdrnd. When a Julia implementation loads the generated sysimg, it will check the host processor for matching CPU capability flags, enabling the highest ISA level possible. Note that the base level (generic) requires the cx16 instruction, which is disabled in some virtualization software and must be enabled for the generic target to be loaded. Alternatively, a sysimg could be generated with the target generic,-cx16 for greater compatibility, however note that this may cause performance and stability problems in some code.

Without the above, Julia is built targeting whatever machine architecture the build system is using. However, we probably should target a broad number of x86_64 architectures rather than ones similar to the virtual machine environment. Here we target the exact same architectures that official Julia targets.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Does that mean we should remove MARCH=core2 from the line below?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. MARCH and JULIA_CPU_TARGET are actually distinct. JULIA_CPU_TARGET specifically targets the LLVM JIT component for building the system image.

https://github.com/JuliaLang/julia/blob/3aea11eea2111e91f33d91926d1fe2f9a71947cb/Make.inc#L774-L781

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the link. Looks like Julia is setting it based on the compiler, so it's better to remove MARCH part.

recipe/build.sh Outdated
fi
# Do this only for x86_64, target platform is not osx-arm64
export JULIA_CPU_TARGET="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)"
https://github.com/JuliaCI/julia-buildbot/blob/ba448c690935fe53d2b1fc5ce22bc60fd1e251a7/master/inventory.py
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the link supposed to be in a comment?

Co-authored-by: Isuru Fernando <isuruf@gmail.com>
@isuruf isuruf added the automerge Merge the PR when CI passes label Sep 7, 2021
@github-actions github-actions bot merged commit 71f2e87 into conda-forge:master Sep 7, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Sep 7, 2021

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

  • linter: passed
  • azure: passed

Thus the PR was passing and merged! Have a great day!

@mkitti
Copy link
Contributor Author

mkitti commented Sep 8, 2021

Oh, I forgot to specify the version of suitesparse for run.

@zklaus
Copy link

zklaus commented Sep 8, 2021

@mkitti addresses this in #124 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Merge the PR when CI passes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants