Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] 413 Client Error: Request Entity Too Large for url #10332

Open
3 of 23 tasks
dotdothu opened this issue Nov 9, 2023 · 4 comments
Open
3 of 23 tasks

[BUG] 413 Client Error: Request Entity Too Large for url #10332

dotdothu opened this issue Nov 9, 2023 · 4 comments
Labels
area/artifacts Artifact stores and artifact logging area/tracking Tracking service, tracking client APIs, autologging bug Something isn't working

Comments

@dotdothu
Copy link
Contributor

dotdothu commented Nov 9, 2023

Issues Policy acknowledgement

  • I have read and agree to submit bug reports in accordance with the issues policy

Where did you encounter this bug?

Other

Willingness to contribute

Yes. I can contribute a fix for this bug independently.

MLflow version

  • Client: 2.7.1
  • Tracking server: 2.7.1

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  • Python version:
  • yarn version, if running the dev UI:

Describe the problem

When I try to log a model above a certain size (25MB worked, probably above 50?) then I face an error.

I use Google Cloud Run, PostgreSQL and Google Cloud Storage

MLFLOW_GCS_UPLOAD_CHUNK_SIZE set for 262144

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>413 Request Entity Too Large</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Request Entity Too Large</h1>
<h2>Your client issued a request that was too large.
</h2>

<h2></h2>
</body></html>

Tracking information

MLflow version: 2.7.1
Tracking URI: https://X/
Artifact URI: mlflow-artifacts:/77/cf85568b0464465dab072417b98eb3bf/artifacts
System information: Windows 10.0.19045
Python version: 3.11.4
MLflow version: 2.7.1
Tracking URI: https://X/
Registry URI: https://X/
Active experiment ID: 77
Active run ID: cf85568b0464465dab072417b98eb3bf
Active run artifact URI: mlflow-artifacts:/77/cf85568b0464465dab072417b98eb3bf/artifacts
MLflow dependencies: 
  Flask: 2.3.3
  Jinja2: 3.1.2
  alembic: 1.12.0
  click: 8.1.7
  cloudpickle: 2.2.1
  databricks-cli: 0.18.0
  docker: 6.1.3
  entrypoints: 0.4
  gitpython: 3.1.40
  google-cloud-storage: 2.12.0
  importlib-metadata: 6.8.0
  markdown: 3.5
  matplotlib: 3.8.0
  numpy: 1.26.1
  packaging: 23.2
  pandas: 2.1.1
  protobuf: 4.24.4
  psutil: 5.9.6
  pyarrow: 13.0.0
  pytz: 2023.3.post1
  pyyaml: 6.0.1
  querystring-parser: 1.2.4
  requests: 2.31.0
  scikit-learn: 1.3.2
  scipy: 1.11.3
  sqlalchemy: 2.0.22
  sqlparse: 0.4.4
  waitress: 2.1.2

Code to reproduce issue

# Works
rf = RandomForestRegressor(n_estimators=10, random_state=42)
rf.fit(X_train, y_train)
signature = mlflow.models.infer_signature(X_test, rf.predict(X_test))
mlflow.sklearn.log_model(rf, artifact_path=model_folder_name, signature=signature)

# Fails
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
signature = mlflow.models.infer_signature(X_test, rf.predict(X_test))
mlflow.sklearn.log_model(rf, artifact_path=model_folder_name, signature=signature)

Stack trace

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\utils\request_utils.py:32, in augmented_raise_for_status(response)
     [31](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:31) try:
---> [32](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:32)     response.raise_for_status()
     [33](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:33) except HTTPError as e:

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\.venv\Lib\site-packages\requests\models.py:1021, in Response.raise_for_status(self)
   [1020](file:///C:/Users/ML/repositories/mlflow-guide-demo/.venv/Lib/site-packages/requests/models.py:1020) if http_error_msg:
-> [1021](file:///C:/Users/ML/repositories/mlflow-guide-demo/.venv/Lib/site-packages/requests/models.py:1021)     raise HTTPError(http_error_msg, response=self)

HTTPError: 413 Client Error: Request Entity Too Large for url: https://ada-mlflow-preprd-juqzwmg2lq-nw.a.run.app/api/2.0/mlflow-artifacts/artifacts/77/30a5817a7c9d4ffe9a50481d7816c9c2/artifacts/models/model.pkl

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
[c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\01_experiment_tracking.ipynb Cell 16 line 2
      [1](vscode-notebook-cell:/c%3A/Users/ML/repositories/mlflow-guide-demo/demo/01_experiment_tracking.ipynb#X51sZmlsZQ%3D%3D?line=0) signature = mlflow.models.infer_signature(X_test, rf.predict(X_test))
----> [2](vscode-notebook-cell:/c%3A/Users/ML/repositories/mlflow-guide-demo/demo/01_experiment_tracking.ipynb#X51sZmlsZQ%3D%3D?line=1) mlflow.sklearn.log_model(rf, artifact_path=model_folder_name, signature=signature)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\sklearn\__init__.py:408, in log_model(sk_model, artifact_path, conda_env, code_paths, serialization_format, registered_model_name, signature, input_example, await_registration_for, pip_requirements, extra_pip_requirements, pyfunc_predict_fn, metadata)
    [328](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:328) @format_docstring(LOG_MODEL_PARAM_DOCS.format(package_name="scikit-learn"))
    [329](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:329) def log_model(
    [330](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:330)     sk_model,
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\sklearn\__init__.py:1'>1</a>;32m   (...)
    [342](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:342)     metadata=None,
    [343](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:343) ):
    [344](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:344)     """
    [345](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:345)     Log a scikit-learn model as an MLflow artifact for the current run. Produces an MLflow Model
    [346](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:346)     containing the following flavors:
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\sklearn\__init__.py:1'>1</a>;32m   (...)
    [406](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:406) 
    [407](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:407)     """
--> [408](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:408)     return Model.log(
    [409](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:409)         artifact_path=artifact_path,
    [410](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:410)         flavor=mlflow.sklearn,
    [411](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:411)         sk_model=sk_model,
    [412](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:412)         conda_env=conda_env,
    [413](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:413)         code_paths=code_paths,
    [414](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:414)         serialization_format=serialization_format,
    [415](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:415)         registered_model_name=registered_model_name,
    [416](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:416)         signature=signature,
    [417](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:417)         input_example=input_example,
    [418](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:418)         await_registration_for=await_registration_for,
    [419](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:419)         pip_requirements=pip_requirements,
    [420](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:420)         extra_pip_requirements=extra_pip_requirements,
    [421](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:421)         pyfunc_predict_fn=pyfunc_predict_fn,
    [422](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:422)         metadata=metadata,
    [423](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:423)     )

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\models\model.py:580, in Model.log(cls, artifact_path, flavor, registered_model_name, await_registration_for, metadata, **kwargs)
    [578](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:578)     _logger.warning(_LOG_MODEL_MISSING_SIGNATURE_WARNING)
    [579](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:579) flavor.save_model(path=local_path, mlflow_model=mlflow_model, **kwargs)
--> [580](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:580) mlflow.tracking.fluent.log_artifacts(local_path, mlflow_model.artifact_path)
    [581](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:581) try:
    [582](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:582)     mlflow.tracking.fluent._record_logged_model(mlflow_model)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\tracking\fluent.py:908, in log_artifacts(local_dir, artifact_path)
    [878](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:878) """
    [879](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:879) Log all the contents of a local directory as artifacts of the run. If no run is active,
    [880](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:880) this method will create a new active run.
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\tracking\fluent.py:1'>1</a>;32m   (...)
    [905](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:905)         mlflow.log_artifacts("data", artifact_path="states")
    [906](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:906) """
    [907](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:907) run_id = _get_or_start_run().info.run_id
--> [908](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:908) MlflowClient().log_artifacts(run_id, local_dir, artifact_path)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\tracking\client.py:1138, in MlflowClient.log_artifacts(self, run_id, local_dir, artifact_path)
   [1094](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1094) def log_artifacts(
   [1095](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1095)     self, run_id: str, local_dir: str, artifact_path: Optional[str] = None
   [1096](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1096) ) -> None:
   [1097](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1097)     """
   [1098](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1098)     Write a directory of files to the remote ``artifact_uri``.
   [1099](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1099) 
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\tracking\client.py:1'>1</a>;32m   (...)
   [1136](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1136)         is_dir: True
   [1137](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1137)     """
-> [1138](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1138)     self._tracking_client.log_artifacts(run_id, local_dir, artifact_path)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\tracking\_tracking_service\client.py:463, in TrackingServiceClient.log_artifacts(self, run_id, local_dir, artifact_path)
    [456](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:456) def log_artifacts(self, run_id, local_dir, artifact_path=None):
    [457](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:457)     """
    [458](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:458)     Write a directory of files to the remote ``artifact_uri``.
    [459](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:459) 
    [460](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:460)     :param local_dir: Path to the directory of files to write.
    [461](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:461)     :param artifact_path: If provided, the directory in ``artifact_uri`` to write to.
    [462](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:462)     """
--> [463](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:463)     self._get_artifact_repo(run_id).log_artifacts(local_dir, artifact_path)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\store\artifact\http_artifact_repo.py:45, in HttpArtifactRepository.log_artifacts(self, local_dir, artifact_path)
     [41](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:41)     artifact_dir = (
     [42](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:42)         posixpath.join(artifact_path, rel_path) if artifact_path else rel_path
     [43](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:43)     )
     [44](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:44) for f in filenames:
---> [45](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:45)     self.log_artifact(os.path.join(root, f), artifact_dir)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\store\artifact\http_artifact_repo.py:31, in HttpArtifactRepository.log_artifact(self, local_file, artifact_path)
     [27](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:27) with open(local_file, "rb") as f:
     [28](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:28)     resp = http_request(
     [29](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:29)         self._host_creds, endpoint, "PUT", data=f, extra_headers=extra_headers
     [30](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:30)     )
---> [31](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:31)     augmented_raise_for_status(resp)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\utils\request_utils.py:35, in augmented_raise_for_status(response)
     [33](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:33) except HTTPError as e:
     [34](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:34)     if response.text:
---> [35](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:35)         raise HTTPError(
     [36](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:36)             f"{e}. Response text: {response.text}", request=e.request, response=e.response
     [37](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:37)         )
     [38](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:38)     else:
     [39](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:39)         raise e

HTTPError: 413 Client Error: Request Entity Too Large for url: https://ada-mlflow-preprd-juqzwmg2lq-nw.a.run.app/api/2.0/mlflow-artifacts/artifacts/77/30a5817a7c9d4ffe9a50481d7816c9c2/artifacts/models/model.pkl. Response text: 
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>413 Request Entity Too Large</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Request Entity Too Large</h1>
<h2>Your client issued a request that was too large.
</h2>
<h2></h2>
</body></html>

Other info / logs

REPLACE_ME

What component(s) does this bug affect?

  • area/artifacts: Artifact stores and artifact logging
  • area/build: Build and test infrastructure for MLflow
  • area/docs: MLflow documentation pages
  • area/examples: Example code
  • area/gateway: AI Gateway service, Gateway client APIs, third-party Gateway integrations
  • area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
  • area/models: MLmodel format, model serialization/deserialization, flavors
  • area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
  • area/projects: MLproject format, project running backends
  • area/scoring: MLflow Model server, model deployment tools, Spark UDFs
  • area/server-infra: MLflow Tracking server backend
  • area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

  • area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
  • area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
  • area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
  • area/windows: Windows support

What language(s) does this bug affect?

  • language/r: R APIs and clients
  • language/java: Java APIs and clients
  • language/new: Proposals for new client languages

What integration(s) does this bug affect?

  • integrations/azure: Azure and Azure ML integrations
  • integrations/sagemaker: SageMaker integrations
  • integrations/databricks: Databricks integrations
@dotdothu dotdothu added the bug Something isn't working label Nov 9, 2023
@github-actions github-actions bot added area/artifacts Artifact stores and artifact logging area/tracking Tracking service, tracking client APIs, autologging labels Nov 9, 2023
@github-actions GitHub Actions
Copy link

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

@AwaisPF
Copy link

AwaisPF commented Apr 1, 2024

any update on this? because i am facing same issue
mlflow version 2.11.1 same version on both client and server

@viktorandreevichmorozov
Copy link

the same for me

@cile98
Copy link

cile98 commented Apr 18, 2024 •
edited

I had previously encountered the same issue. In my case it was caused by proxy-body-size nginx parameter. I've set it to 256m and it fixed the problem (sizes of my models were around 40-50 MB)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/artifacts Artifact stores and artifact logging area/tracking Tracking service, tracking client APIs, autologging bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dotdothu @cile98 @viktorandreevichmorozov @AwaisPF