Restricting cornac to 1.15.1 for issue with 1.15.4 #1934

miguelgfierro · 2023-06-01T16:24:06Z

Description

See #1933

Related Issues

References

Checklist:

I have followed the contribution guidelines and code style for this project.
I have added tests covering my contributions.
I have updated the documentation accordingly.
This PR is being made to staging branch and not to main branch.

miguelgfierro · 2023-06-01T16:41:06Z

mmmm it seems it is not taking the changing, it still uses cornac 1.15.4, instead of 1.15.1.

Collecting cornac<2,>=1.1.2
  Downloading cornac-1.15.4.tar.gz (7.9 MB)
     â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”� 7.9/7.9 MB 121.1 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'

failed
Pip subprocess error:
  error: subprocess-exited-with-error
  
  Ã— python setup.py egg_info did not run successfully.
  â”‚ exit code: 1
  â•°â”€> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 36, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-pfz_52us/cornac_4b74720c302545d1abaa7375b4556cb1/setup.py", line 32, in <module>
          import numpy as np
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

Why don't you obey me?

o_o

The problem is that we are installing the pip library pip install recommenders instead of the source code pip install .

    # install extra dependencies
    if add_gpu_dependencies and add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        conda_dep.add_pip_package("recommenders[dev,examples,spark,gpu]")
    elif add_gpu_dependencies:
        conda_dep.add_pip_package("recommenders[dev,examples,gpu]")
    elif add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        conda_dep.add_pip_package("recommenders[dev,examples,spark]")
    else:
        conda_dep.add_pip_package("recommenders[dev,examples]")

miguelgfierro · 2023-06-02T06:29:59Z

Tried in submit_groupwise_azureml_pytest.py

    # install extra dependencies
    if add_gpu_dependencies and add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        conda_dep.add_pip_package("[dev,examples,spark,gpu]")
    elif add_gpu_dependencies:
        conda_dep.add_pip_package("[dev,examples,gpu]")
    elif add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        conda_dep.add_pip_package("[dev,examples,spark]")
    else:
        conda_dep.add_pip_package("[dev,examples]")

error:

Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/azureml-envs/azureml_8fe95e22b208ec42145257f7dcae20e3/bin/python', '-m', 'pip', 'install', '-U', '-r', '/azureml-environment-setup/condaenv.xes2msz1.requirements.txt', '--exists-action=b']
Pip subprocess output:

failed
Pip subprocess error:
ERROR: Directory '.[dev,examples,gpu]' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

miguelgfierro · 2023-06-02T19:30:15Z

Tried:

    # install extra dependencies
    if add_gpu_dependencies and add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        # conda_dep.add_pip_package("recommenders[dev,examples,spark,gpu]")
        run_azuremlcompute.script = ["pip", "install", ".[dev,examples,spark,gpu]"]
    elif add_gpu_dependencies:
        # conda_dep.add_pip_package("recommenders[dev,examples,gpu]")
        run_azuremlcompute.script = ["pip", "install", ".[dev,examples,gpu]"]
    elif add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        # conda_dep.add_pip_package("recommenders[dev,examples,spark]")
        run_azuremlcompute.script = ["pip", "install", ".[dev,examples,spark]"]

    else:
        # conda_dep.add_pip_package("[dev,examples]")
        run_azuremlcompute.script = ["pip", "install", ".[dev,examples]"]

But I get the same error as when it install with pip install recommenders. It should download cornac 1.15.1, but it downloads 1.15.4

Collecting cornac<2,>=1.1.2
  Downloading cornac-1.15.4.tar.gz (7.9 MB)
     â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”�â”� 7.9/7.9 MB 107.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'

failed
Pip subprocess error:
  error: subprocess-exited-with-error
  
  Ã— python setup.py egg_info did not run successfully.
  â”‚ exit code: 1
  â•°â”€> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 36, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-llx3g59p/cornac_544ed4953ab44044905d844e3babd685/setup.py", line 32, in <module>
          import numpy as np
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

miguelgfierro · 2023-06-02T19:45:14Z

@pradnyeshjoshi I'm trying to install the local recommenders pip install . instead of the pip version (see messages above) using RunConfiguration() class of AzureML. But I can't find a solution. Do you have experience with it?
Maybe @loomlike has experience with this part of AzureML?

miguelgfierro · 2023-06-02T20:08:29Z

tried with command:

    # install extra dependencies
    if add_gpu_dependencies and add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        # conda_dep.add_pip_package("recommenders[dev,examples,spark,gpu]") # Not working, it installs the pip package instead of the local wheel
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples,spark,gpu]"] # not working
        run_azuremlcompute.command = ["pip", "install", ".[dev,examples,spark,gpu]"] 
    elif add_gpu_dependencies:
        # conda_dep.add_pip_package("recommenders[dev,examples,gpu]") # Not working, it installs the pip package instead of the local wheel
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples,gpu]"] # not working
        run_azuremlcompute.command = ["pip", "install", ".[dev,examples,gpu]"]
    elif add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        # conda_dep.add_pip_package("recommenders[dev,examples,spark]") # Not working, it installs the pip package instead of the local wheel
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples,spark]"] # not working
        run_azuremlcompute.command = ["pip", "install", ".[dev,examples,spark]"]

    else:
        # conda_dep.add_pip_package("[dev,examples]")
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples]"]
        run_azuremlcompute.command = ["pip", "install", ".[dev,examples]"]

I got the error: Both Script and Command cannot be specified with the request

miguelgfierro · 2023-06-02T20:12:16Z

Tried interpreter options:

    # install extra dependencies
    if add_gpu_dependencies and add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        # conda_dep.add_pip_package("recommenders[dev,examples,spark,gpu]") # Not working, it installs the pip package instead of the local wheel
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples,spark,gpu]"] # not working
        # run_azuremlcompute.command = ["pip", "install", ".[dev,examples,spark,gpu]"]  # Error: Both Script and Command cannot be specified with the request
        run_azuremlcompute.environment.python.interpreter_options = "-m venv .venv && source .venv/bin/activate && pip install -e .[dev,examples,spark,gpu]"    
    elif add_gpu_dependencies:
        # conda_dep.add_pip_package("recommenders[dev,examples,gpu]") # Not working, it installs the pip package instead of the local wheel
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples,gpu]"] # not working
        # run_azuremlcompute.command = ["pip", "install", ".[dev,examples,gpu]"] # Error: Both Script and Command cannot be specified with the request
        run_azuremlcompute.environment.python.interpreter_options = "-m venv .venv && source .venv/bin/activate && pip install -e .[dev,examples,gpu]"
    elif add_spark_dependencies:
        conda_dep.add_channel("conda-forge")
        conda_dep.add_conda_package(conda_pkg_jdk)
        # conda_dep.add_pip_package("recommenders[dev,examples,spark]") # Not working, it installs the pip package instead of the local wheel
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples,spark]"] # not working
        # run_azuremlcompute.command = ["pip", "install", ".[dev,examples,spark]"] # Error: Both Script and Command cannot be specified with the request
        run_azuremlcompute.environment.python.interpreter_options = "-m venv .venv && source .venv/bin/activate && pip install -e .[dev,examples,spark]"
    else:
        # conda_dep.add_pip_package("[dev,examples]")
        # run_azuremlcompute.script = ["pip", "install", ".[dev,examples]"] # not working
        # run_azuremlcompute.command = ["pip", "install", ".[dev,examples]"] # Error: Both Script and Command cannot be specified with the request
        run_azuremlcompute.environment.python.interpreter_options = "-m venv .venv && source .venv/bin/activate && pip install -e .[dev,examples]"

got the error: AttributeError: <class 'azureml.core.environment.PythonSection'> has no attribute interpreter_options

miguelgfierro · 2023-06-05T15:31:10Z

Tried to set exist_ok to False

    # install local version of recommenders on AzureML compute using .whl file
    whl_url = run_azuremlcompute.environment.add_private_pip_wheel(
        workspace=workspace,
        file_path=reco_wheel_path,
        exist_ok=False,
    )

I get this error:


"message": "UserError: Resource Conflict: ArtifactId Environment/azureml-private-packages/recommenders-1.1.1-py3-none-any.whl already exists."
Traceback (most recent call last):
  File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 468, in <module>
    run_config = create_run_config(
  File "tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py", line 192, in create_run_config
    whl_url = run_azuremlcompute.environment.add_private_pip_wheel(
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/azureml/core/environment.py", line 1432, in add_private_pip_wheel
    raise UserErrorException(error_msg)
azureml.exceptions._azureml_exception.UserErrorException: UserErrorException:
	Message: 
A wheel with the name recommenders-1.1.1-py3-none-any.whl already exists.
Please make sure recommenders-1.1.1-py3-none-any.whl is the private wheel you would like to add to the workspace.
If you would like to have it be a no-op instead of throwing an exception for duplicate entries,
please set exist_ok=True
	InnerException None
	ErrorResponse

This won't solve the problem because still we need to install the deps that we define in the local setup.py instead of the ones defined in the recommenders pip.

miguelgfierro · 2023-06-06T14:18:30Z

Trying to build a dockerfile from scratch:

Only CPU for now

    run_azuremlcompute = RunConfiguration()
    run_azuremlcompute.target = cpu_cluster
    run_azuremlcompute.environment.docker.use_docker = True
    # run_azuremlcompute.environment.docker.base_image = docker_proc_type
    
    dockerfile = """
    FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04

    ARG HOME
    ARG VIRTUAL_ENV
    ENV HOME="${HOME}"
    WORKDIR ${HOME}

    RUN apt-get update && \
    apt-get install -y curl build-essential

    # Install Anaconda
    ARG ANACONDA="https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"
    RUN curl ${ANACONDA} -o anaconda.sh && \
    /bin/bash anaconda.sh -b -p conda && \
    rm anaconda.sh && \
    echo ". ${HOME}/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
    echo "conda activate base" >> ~/.bashrc ; fi

    # Install Python packages
    RUN conda install python=3.7
    RUN pip install .[dev,examples]

    """
    run_azuremlcompute.environment.docker.base_dockerfile = dockerfile

simonzhaoms · 2023-06-07T02:30:57Z

tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py

+    run_azuremlcompute.environment.docker.enabled = True
+    # run_azuremlcompute.environment.docker.base_image = docker_proc_type
+
+    dockerfile = """


I think it is more reliable to use CondaDependencies to set up the Conda env than dockerfile

@miguelgfierro I see why you tried to use dockerfile here. I think it should be possible to use CondaDependencies or the like to install recommenders locally. I'll do a research.

miguelgfierro · 2023-06-08T10:40:10Z

This is awesome @simonzhaoms

Restricting cornac to 1.15.1

dc64de0

miguelgfierro requested review from gramhagen, anargyri, loomlike, wutaomsft and simonzhaoms as code owners June 1, 2023 16:24

miguelgfierro mentioned this pull request Jun 1, 2023

Remove azure cosmos #1935

Merged

4 tasks

miguelgfierro added 3 commits June 2, 2023 08:07

restrict cornac

584c4ec

Clarification cornac

cc8edd1

Install the development environment instead of the pip package

fce46cc

miguelgfierro added 6 commits June 2, 2023 08:30

Install the development environment instead of the pip package

4938e38

Install the development environment instead of the pip package

7a69fba

Install the development environment instead of the pip package

21460e5

Install the development environment instead of the pip package

71bcd8f

Install the development environment instead of the pip package

643d81b

Try with interpreter options

6960cb5

miguelgfierro added 2 commits June 5, 2023 17:21

cornac

e18a4be

WIP

81c2bce

miguelgfierro added 5 commits June 6, 2023 09:47

Try building dockerfile

21b519b

Error with spark and gpu excluded tests

a4adec9

Error with spark and gpu excluded tests

cdaeb7a

Error with spark and gpu excluded tests

5f80758

Error with spark and gpu excluded tests

1c73cfb

miguelgfierro mentioned this pull request Jun 6, 2023

Add support for Python 3.10 and 3.11 #1937

Merged

4 tasks

simonzhaoms reviewed Jun 7, 2023

View reviewed changes

simonzhaoms added 2 commits June 7, 2023 16:13

Try to install the local wheel of recommenders

be68442

Change pip install scheme

67c6d25

miguelgfierro mentioned this pull request Jun 7, 2023

Remove bottleneck dep #1938

Merged

4 tasks

simonzhaoms added 2 commits June 8, 2023 10:33

Test normal usage of Environment.add_private_pip_wheel()

754e428

Install recommenders from GitHub

28cdedc

simonzhaoms self-requested a review June 8, 2023 08:55

Clean up

caf7192

simonzhaoms approved these changes Jun 8, 2023

View reviewed changes

miguelgfierro merged commit 58753b6 into staging Jun 8, 2023
26 checks passed

miguelgfierro deleted the miguel/bug_cornac_numpy branch June 8, 2023 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restricting cornac to 1.15.1 for issue with 1.15.4 #1934

Restricting cornac to 1.15.1 for issue with 1.15.4 #1934

miguelgfierro commented Jun 1, 2023

miguelgfierro commented Jun 1, 2023 •

edited

miguelgfierro commented Jun 2, 2023 •

edited

miguelgfierro commented Jun 2, 2023 •

edited

miguelgfierro commented Jun 2, 2023 •

edited

miguelgfierro commented Jun 2, 2023

miguelgfierro commented Jun 2, 2023

miguelgfierro commented Jun 5, 2023 •

edited

miguelgfierro commented Jun 6, 2023

simonzhaoms Jun 7, 2023

simonzhaoms Jun 7, 2023 •

edited

miguelgfierro commented Jun 8, 2023

Restricting cornac to 1.15.1 for issue with 1.15.4 #1934

Restricting cornac to 1.15.1 for issue with 1.15.4 #1934

Conversation

miguelgfierro commented Jun 1, 2023

Description

Related Issues

References

Checklist:

miguelgfierro commented Jun 1, 2023 • edited

miguelgfierro commented Jun 2, 2023 • edited

miguelgfierro commented Jun 2, 2023 • edited

miguelgfierro commented Jun 2, 2023 • edited

miguelgfierro commented Jun 2, 2023

miguelgfierro commented Jun 2, 2023

miguelgfierro commented Jun 5, 2023 • edited

miguelgfierro commented Jun 6, 2023

simonzhaoms Jun 7, 2023

Choose a reason for hiding this comment

simonzhaoms Jun 7, 2023 • edited

Choose a reason for hiding this comment

miguelgfierro commented Jun 8, 2023

miguelgfierro commented Jun 1, 2023 •

edited

miguelgfierro commented Jun 2, 2023 •

edited

miguelgfierro commented Jun 2, 2023 •

edited

miguelgfierro commented Jun 2, 2023 •

edited

miguelgfierro commented Jun 5, 2023 •

edited

simonzhaoms Jun 7, 2023 •

edited