add _openssl-syslib-check package

[h-vetinari]

Checklist

• Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml"
• License file is packaged (see here for an example)
• Source is from official source
• Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged)
• If static libraries are linked in, the license of the static library is packaged.
• Build number is 0
• A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details)
• GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there

Out of the discussion in conda-forge/cryptography-feedstock#37 and conda-forge/openssl-feedstock#56, @mingwandroid suggested to introduce a package that both the openssl- & cryptography-feedstock can depend on, so we don't have a mess of activation scripts needing to cover all possible deployment scenarios.

This is a first draft, but comes with a test suite at least. I added some old dlls for that purpose, not sure what the opinion is on that.

I added both @isuruf and @mingwandroid as maintainers, since they were by far the most active/knowledgeable in this discussion. If this assumption is wrong, I'm happy to revert.

Still missing:

• write activate.sh and enable testing for it.
• figure out a way to execute run_test.py with enough permissions to change stuff in C:/Windows/System32 (see this comment)
• ensure recipe gets tested with correct python version per build (see this comment)

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/_openssl-syslib-check) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipes/_openssl-syslib-check:

• License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/_openssl-syslib-check) and found it was in an excellent condition.

[h-vetinari]

Is there a way to run the tests as root?

Since it was possible to delete dlls from C:/Windows/System32 (e.g. conda-forge/conda-forge-ci-setup-feedstock#84), I thought this could be done at testing time as well?

Traceback (most recent call last):
  File "C:\Miniconda\conda-bld\_openssl-syslib-check_1588379506085\test_tmp\run_test.py", line 45, in <module>
    shutil.copyfile('TESTING_ONLY_DO_NOT_USE/1.1.1d-libssl.dll', LIBSSL_PATH)
  File "C:\Miniconda\conda-bld\_openssl-syslib-check_1588379506085\_test_env\lib\shutil.py", line 261, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: 'C:/Windows/System32/libssl-1_1-x64.dll'

For the moment I wrapped that part in a try-except clause, but the activation script should IMO really be tested in all variants.

[isuruf]

I don't think this is a good idea as this adds cycles in the dependency graph.

[h-vetinari]

I don't think this is a good idea as this adds cycles in the dependency graph.

No, this package doesn't depend on anything. And openssl / cryptography could depend on it just on windows.

[h-vetinari]

Edit: nevermind, switched the order, so this is not relevant anymore

I wanted to use a trick from the blas-feedstock to install cryptography during the test, but I can't seem to get it to work.

Traceback (most recent call last):
  File "C:\Miniconda\conda-bld\_openssl-syslib-check_1588381167733\test_tmp\run_test.py", line 55, in <module>
    subprocess.call(['conda.exe install -c conda-forge --yes --quiet cryptography'])
[...]
FileNotFoundError: [WinError 2] The system cannot find the file specified

[h-vetinari]

@conda-forge/staged-recipes
Is there any way to run the tests as an admin? I need to copy some outdated libs to C:/Windows/System32/ to fully test all scenarios of the activation script (this is harmless, cf. conda-forge/conda-forge-ci-setup-feedstock#84).

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/_openssl-syslib-check) and found some lint.

Here's what I've got...

For recipes/_openssl-syslib-check:

• noarch packages can't have selectors. If the selectors are necessary, please remove noarch: python.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/_openssl-syslib-check) and found it was in an excellent condition.

[h-vetinari]

@conda-forge/staged-recipes
So conda-smithy says the recipe can't be noarch because of the # [not win] selector, yet while the build process says it's building:

Resolved dependencies, will be built in the following order:
    _openssl-syslib-check-1.0.0-python_3.6.* *_cpython-on-win-64
    _openssl-syslib-check-1.0.0-python_3.7.* *_cpython-on-win-64

the test environment actually installs python 3.8 in both cases.

This recipe must not depend on python, but I'd like to be able to test the correct python version nevertheless (because different python versions have different builds regarding CONDA_DLL_SEARCH_MODIFICATION_ENABLE, cf. conda-forge/python-feedstock#345) - how can I achieve that?

[h-vetinari]

@mingwandroid
This implements the idea you voiced on the openssl-PR. It's IMO ready for a round of review - I'd still need to write activate.sh, but thankfully that'd be just a (much easier) translation of the .bat version. There are also two open points that I don't have an answer for yet (see OP).

[h-vetinari]

PS. As noted here, there's an argument to be made that the activation scripts could directly live in the openssl-feedstock, and probe for cryptography without depending on it. The only downside I see is that I don't know how I could explicitly test with the outdated DLLs as I'm trying to do here (I've tested that it works, modulo the permission errors on azure).

[h-vetinari]

@mingwandroid
What do you think about the approach here - both the general structure of activate.bat and the tests for it?

Strictly speaking a test suite for the activation scripts is not necessary, but I thought it's a prudent thing considering the importance of the packages (if we say goodbye to the tests that need elevated privileges, I could implement this in the openssl-feedstock right away).

[mingwandroid]

I like it, yes, many thanks. My only concern is a slight overhead added at activate time, but I guess we'll see if that's a big problem. It's certainly better than the alternative! It might be prudent to have a very-early-out env. var check to mitigate this? CONDA_SKIP_OPENSSL_DLL_CHECK or something?

[h-vetinari]

@mingwandroid
Good to hear!

I have already added an early out:

:: early exit in case no syslibs are found
if "%HAS_SYS_LIBS%"=="F" (
  exit /b 0
)

I also thought about an early-out env-var, but then I guess that 2 file-exist-checks are probably almost unnoticeably quick? If I'm wrong about that, an even earlier out is easy to add of course.

Do you have some advice if/how run_test.py could be run with elevated privileges? And is there a jinja-variable to template over the python-verison in the test-env? Finally, assuming this should just live in openssl directly, would there be a way to pass non-source files from the recipe to the test environment (thinking about the outdated DLLs here)?

[h-vetinari]

PS. Besides speed, the CONDA_SKIP_OPENSSL_DLL_CHECK would serve a potentially much more important usability-point as well - silencing the warning, for those who don't wanna see it.

[h-vetinari]

@isuruf
Thanks for jumping in and helping out! Is there any way we could execute run_test.py with elevated permissions? I mean, the conda-forge-ci-setup definitely runs with those privileges, so I'm guessing it should not be completely impossible to achieve for the testing either?

[h-vetinari]

@mingwandroid @isuruf

This is more or less ready to be added into the openssl-PR, but I still have the open question if there is a way to execute run_test.py with elevated privileges somehow?

Correction: might still need a bit of work for piping the output of activate.sh back to run_test.py. I've also thought about actually assert-ing the content of the message, which would be easy to add.

[h-vetinari]

Ping @mingwandroid @isuruf

[stale]

Hi friend!

We really, really, really appreciate that you have taken the time to make a PR on conda-forge/staged-recipes! conda-forge only exists because people like you donate their time to build and maintain conda recipes for use by the community.

In an effort to maintain this repository and increase the signal-to-noise for open PRs, the maintainers of staged-recipes close excessively old PRs after six months. This PR will remain open for another month, and then will be closed.

If you'd like to keep it open, please comment/push and we will be happy to oblige! Note that very old PRs will likely need to be rebased on master so that they can be rebuilt with the most recent CI scripts. If you have any trouble, or we missed reviewing this PR in the first place (sorry!), feel free to ping the team using a special command in a comment on the PR to get the attention of the staged-recipes team.

Cheers and thank you for contributing to this community effort!

[h-vetinari]

Let's keep this open a bit more. Still planning to come back to this in some form

[stale]

Hi friend!

We really, really, really appreciate that you have taken the time to make a PR on conda-forge/staged-recipes! conda-forge only exists because people like you donate their time to build and maintain conda recipes for use by the community.

In an effort to maintain this repository and increase the signal-to-noise for open PRs, the maintainers of staged-recipes close excessively old PRs after six months. This PR will remain open for another month, and then will be closed.

If you'd like to keep it open, please comment/push and we will be happy to oblige! Note that very old PRs will likely need to be rebased on master so that they can be rebuilt with the most recent CI scripts. If you have any trouble, or we missed reviewing this PR in the first place (sorry!), feel free to ping the team using a special command in a comment on the PR to get the attention of the staged-recipes team.

Cheers and thank you for contributing to this community effort!

[h-vetinari]

Not (completely) stale

[xhochy]

Not (completely) stale

What's the state here? Do you plan to work more on this or should I do a review to get this forward?

[h-vetinari]

Hey @xhochy, thanks for taking an interest here!

There were a few open questions for which I never got an answer - mainly revolving around testing. E.g. to test the correct behaviour in all (or even most) variants, we'd have to delete/replace system libraries in CI, which requires running the test with elevated privileges.

Also, I had considered writing some tests for the activation-log output, but never got around to that due to not receiving other feedback - just reread the thread and most things are still current/open.

For context, this PR started here, and then Ray had suggested:

Why don't we make a common package for this check that both openssl and anything else can depend on?

[stale]

Hi friend!

We really, really, really appreciate that you have taken the time to make a PR on conda-forge/staged-recipes! conda-forge only exists because people like you donate their time to build and maintain conda recipes for use by the community.

In an effort to maintain this repository and increase the signal-to-noise for open PRs, the maintainers of staged-recipes close excessively old PRs after six months. This PR will remain open for another month, and then will be closed.

If you'd like to keep it open, please comment/push and we will be happy to oblige! Note that very old PRs will likely need to be rebased on main so that they can be rebuilt with the most recent CI scripts. If you have any trouble, or we missed reviewing this PR in the first place (sorry!), feel free to ping the team using a special command in a comment on the PR to get the attention of the staged-recipes team.

Cheers and thank you for contributing to this community effort!

[stale]

Hi again! About a month ago, we commented on this PR saying it would be closed in another month if it was still inactive. It has been a month and so now it is being closed. Thank you so much for making it in the first place and contributing to the community project that is conda-forge. If you'd like to reopen this PR, please feel free to do so at any time!

Cheers and have a great day!