[CLI]: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url

[itskevinwang]

Describe the bug

I began encountering this issue around two weeks ago while using a cluster to run my sweeps. I did not update my code or any packages including wandb, it was working fine one day and then the next day it wasn't. I can successfully use the wandb login command and initialize my sweep, but each run fails due to a requests.exceptions.HTTPError: 400 Client Error: Bad Request for url error pointing towards a storage.googleapi link with the following contents:

https://media.discordapp.net/attachments/945038637798674472/956925793999659008/Screen_Shot_2022-03-25_at_7.06.14_AM.png

I'm unsure if this is a cluster-side error or a firewall blocking wandb, but have reached out to cluster support as well.

I also encountered a SSL error and had to add the following lines to my code, but I'm not sure if it is related:

ssl._create_default_https_context = ssl._create_unverified_context

wandb: logging graph, to disable use `wandb.watch(log_graph=False)`
Traceback (most recent call last):
  File "train_model.py", line 152, in <module>
    main()
  File "train_model.py", line 75, in main
    trainer = Trainer.from_argparse_args(args, callbacks=callbacks, logger=loggers, deterministic=False)
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1923, in from_argparse_args
    return from_argparse_args(cls, args, **kwargs)
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/pytorch_lightning/utilities/argparse.py", line 65, in from_argparse_args
    return cls(**trainer_kwargs)
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 38, in insert_env_defaults
    return fn(self, **kwargs)
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 510, in __init__
    self.on_init_start()
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/pytorch_lightning/trainer/callback_hook.py", line 58, in on_init_start
    callback.on_init_start(self)
  File "/gpfsm/ccds01/nobackup/people/kswang/semanticSegmentation/IceClassifier/training/callbacks/log_examples.py", line 35, in on_init_start
    output_dir = self.__fetch_examples()
  File "/gpfsm/ccds01/nobackup/people/kswang/semanticSegmentation/IceClassifier/training/callbacks/log_examples.py", line 42, in __fetch_examples
    artifact_dir = artifact.download(root=dataset_dir_root)
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/wandb/apis/public.py", line 3765, in download
    pool.map(partial(self._download_file, root=dirpath), manifest.entries)
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/wandb/apis/public.py", line 3855, in _download_file
    return self.get_path(name).download(root)
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/wandb/apis/public.py", line 3284, in download
    cache_path = manifest.storage_policy.load_file(
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/wandb/sdk/wandb_artifacts.py", line 892, in load_file
    response.raise_for_status()
  File "/home/kswang/.conda/envs/semanticsegmentation/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://storage.googleapis.com/wandb-artifacts-prod/semanticsegmentation/IceClassificationGoodlight/wandb_artifacts/16398352/100182298/fb8c37e564dade40c242da751772df11?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=wandb-production%40appspot.gserviceaccount.com%2F20220325%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220325T141134Z&X-Goog-Expires=59&X-Goog-Signature=30995b3393eec4c101a986df5e8732aa641bcfa10c21b0114b68e62d036362962108aef4e5da8a2d5e364ecba1295f138a5f4f012ba70ff921721ff393cd55c3d2d0dd2e15867183862bdc1376c365df07f335cb42cff0aa419376808cc9f8afe358627ac03caeca44c3b4fefeaf00d24b789c7dfe7b1bc0c3246e8f2165c216c049eb1d7b4e1b89bd768c966a188942e358e8236621cdc6a1f68df808885fab1314dd9dc452d70b1724d4c15a9f383139c8fc2493efa0240440f53e91193bc12508797ca8d45f82a5ca154b28d785c48fc95e84599dd7eaf8798d61bae9cfebb60837015f7b8df6a1ce4c986e391cb60f8d422f2821a97834601707ef0e236b&X-Goog-SignedHeaders=host&response-content-disposition=attachment

Additional Files

No response

Environment

WandB version: 0.12.11

OS: CentOS

Python version: 3.8

Versions of relevant libraries:

albumentations 1.1.0
certifi 2021.10.8
google 3.0.0
google-auth 1.35.0
google-auth-oauthlib 0.4.6
matplotlib 3.4.3
networkx 2.6.3
numpy 1.21.2
oauthlib 3.1.1
opencv-python-headless 4.5.4.58
pytorch-lightning 1.5.0
requests 2.26.0
requests-oauthlib 1.3.0
scikit-image 0.18.3
scikit-learn 1.0.1
scipy 1.7.1
tensorboard 2.2.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
torch 1.10.0
torchaudio 0.10.0
torchmetrics 0.6.0
torchvision 0.11.1
urllib3 1.26.7
wandb 0.12.11

Additional Context

No response

[itskevinwang]

debug.log File

2022-03-25 10:14:19,964 INFO    MainThread:48415 [wandb_setup.py:_flush():75] Loading settings from /home/kswang/.config/wandb/settings
2022-03-25 10:14:19,965 INFO    MainThread:48415 [wandb_setup.py:_flush():75] Loading settings from wandb/settings
2022-03-25 10:14:19,965 WARNING MainThread:48415 [wandb_setup.py:_flush():75] Unknown environment variable: WANDB_CACHE_DIR
2022-03-25 10:14:19,965 INFO    MainThread:48415 [wandb_setup.py:_flush():75] Loading settings from environment variables: {'api_key': $
2022-03-25 10:14:19,965 INFO    MainThread:48415 [wandb_setup.py:_flush():75] Inferring run settings from compute environment: {'progra$
2022-03-25 10:14:19,965 INFO    MainThread:48415 [wandb_setup.py:_flush():75] Applying settings from <Settings {'_args': None, '_cli_on$
2022-03-25 10:14:19,966 INFO    MainThread:48415 [wandb_init.py:_log_setup():405] Logging user logs to /gpfsm/ccds01/nobackup/people/ks$
2022-03-25 10:14:19,967 INFO    MainThread:48415 [wandb_init.py:_log_setup():406] Logging internal logs to /gpfsm/ccds01/nobackup/peopl$
2022-03-25 10:14:19,968 INFO    MainThread:48415 [wandb_init.py:init():439] calling init triggers
2022-03-25 10:14:19,968 INFO    MainThread:48415 [wandb_init.py:init():442] wandb.init called with sweep_config: {'backbone_model': 'fc$
config: {}
2022-03-25 10:14:19,969 INFO    MainThread:48415 [wandb_init.py:init():492] starting backend
2022-03-25 10:14:19,969 INFO    MainThread:48415 [backend.py:_multiprocessing_setup():99] multiprocessing start_methods=fork,spawn,fork$
2022-03-25 10:14:20,052 INFO    MainThread:48415 [backend.py:ensure_launched():219] starting backend process...
2022-03-25 10:14:20,074 INFO    MainThread:48415 [backend.py:ensure_launched():224] started backend process with pid: 48483
2022-03-25 10:14:20,078 INFO    MainThread:48415 [wandb_init.py:init():501] backend started and connected
2022-03-25 10:14:20,087 INFO    MainThread:48415 [wandb_run.py:_config_callback():992] config_cb None None {'backbone_model': 'fcn_resn$
2022-03-25 10:14:20,090 INFO    MainThread:48415 [wandb_init.py:init():565] updated telemetry
2022-03-25 10:14:20,092 INFO    MainThread:48415 [wandb_init.py:init():596] communicating run to backend with 30 second timeout```

[lesliewandb]

Can you tell me what you are doing when you run into this error? You had mentioned that wandb.login works, can you send in a code snippet that you think creates this issue?

[exalate-issue-sync]

Leslie commented:
Hi, do you still need help with this issue?

[itskevinwang]

Hi, this issue was resolved when the cluster underwent a network update and I reverted back to a prior version of wandb.

[exalate-issue-sync]

Leslie commented:
Thank you for the update! Can you tell me what version of wandb you had reverted to please?

[exalate-issue-sync]

Leslie commented:
Hi, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

[yasherali]

hi i'm facing this same error
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://onesignal.com/api/v1/notifications

but don't know how to counter it in pythonanywhere, if anyone would help me out that would be great