It seems *api_server* is misconfigured. #98
Comments
|
Hi @Zeko403, Can you verify trains-server has been successfully started? The easiest way to do that is to open your browser, go to the trains-webserver url (in your case, |
|
The webserver works fine by the looks of it, I can access it and type my username there, also I can access Edit: Output from the console: |
|
Hi @Zeko403 Could you delete the $ trains initIt should not only configure the all the servers, but it will also try to validate your credentials with the trains-server. Let me know if it helped. |
|
Hi @bmartinn OK so I tried deleting the trains.conf file (I'm on Windows so the trains.conf file was in C:\Users\Name\trains.conf). Then I ran the trains-init which resulted in following This was the last line of trains-apiserver after starting the docker: Then after I ran the trains-init showing that it actually is responsive: After I pasted the code from the generated access key: TRAINS SDK setup process Please create new trains credentials through the profile page in your trains web app (e.g. http://localhost:8080/profile) Paste copied configuration here: TRAINS Hosts configuration: Verifying credentials ... After which I put the same key and secret in and got the same error api_server is misconfigured. It seems there is some communication error but I cannot put my finger on it... |
|
Hi @Zeko403 , my apologies for the delay, this thread somehow slipped away from me. It seems there is a bug in the message print, it printed the default "https://demoapi.trains.allegro.ai" instead of the But the real question is why the credentials failed to verify with the API server on This is actually consistent with the original post where it crashed with: line 545, in _do_refresh_token
return resp["data"]["token"]
KeyError: 'data'The reason the I suspect it might be the mongo docker. Could you verify that all docker services are working correctly ? Please execute: It should list all 6 containers with equal uptime (If one is misbehaving, it will either not be listed or uptime will be a few seconds since it will constantly restart): CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2d7820ab0fd3 allegroai/trains:latest "/opt/trains/wrapper…" 6 weeks ago Up 17 minutes 8008/tcp, 8080-8081/tcp, 0.0.0.0:8080->80/tcp trains-webserver
9fc6acc64930 allegroai/trains:latest "/opt/trains/wrapper…" 6 weeks ago Up 17 minutes 0.0.0.0:8008->8008/tcp, 8080-8081/tcp trains-apiserver
8d944ba0ac80 redis:5.0 "docker-entrypoint.s…" 6 weeks ago Up 17 minutes 0.0.0.0:6379->6379/tcp trains-redis
b6af7111e3eb allegroai/trains:latest "/opt/trains/wrapper…" 6 weeks ago Up 17 minutes 8008/tcp, 8080/tcp, 0.0.0.0:8081->8081/tcp trains-fileserver
4f9b7db06115 docker.elastic.co/elasticsearch/elasticsearch:5.6.16 "/bin/bash bin/es-do…" 2 months ago Up 17 minutes 0.0.0.0:9200->9200/tcp, 9300/tcp trains-elastic
3ed89332e232 mongo:3.6.5 "docker-entrypoint.s…" 2 months ago Up 17 minutes 0.0.0.0:27017->27017/tcp trains-mongo |
|
Hi again @bmartinn thanks for helping out The output i get is pretty similar to what you expected: |
|
Hi @Zeko403, This is very strange... It seems the mongodb container is up and running... The first call should simply display the server version and build, while the second one attempts to obtain a token based on the user credentials you shared in a previous post on this thread. |
|
EDIT: To answer your questions : and Disregard the comment below, I figured out that I ran the cmd.exe with admin rights and then it did not ignore the proxy rules: Hi @bmartinn and thanks for still helping out. I tried calling curl for both links but I get no response whatsoever, then I updated curl to the newest version, still the same. So I tried adding "-v" in addition So I am not sure if there is something wrong because I am using a proxy, I did set up the proxy in the docker as described. Interestingly, when I just try to look up
|
|
Hi @Zeko403 , I think you just pointed to the problem, the proxy server configuration is not picked unless you are running with Admin privileges. Could you run the python script with elevated Admin privilege? (You can start an elevated Admin p.s. |
|
Hi @bmartinn I tried running Docker, cmd in addition, running trains-init also with admin rights gives the same error as before the Nowhere have I configured the TRAINS API server to point to https://demoapi.trains.allegro.ai, so I don't know where this link is coming from. I get the feeling that server api keeps looking at the https://demoapi.trains.allegro.ai instead of localhost:8080 to communicate with the webserver even when I am using the I generated a new key on webserver and then called Why is there a message Port 8008 is the api port. Replacing 8080 with 8008 for Web application if I added the web_server: http://localhost:8080 and api_server: http://localhost:8008 ? |
Apologies @Zeko403 , this was a bug with the information printed in Here is a recap of what I understand so far:
Is the above correct? |
When I call I will try updating trains and report if there are any changes EDIT: |
|
I suspect this is the proxy forwarding ... import requests
requests.get('http://localhost:8008/server.info').text |
|
it seems you are most likely right, the output is pretty long, but it is a response from the proxy server saying that the request is blocked due to invalid server IP. |
|
hmmm, accessing import requests
requests.get('http://127.0.0.1:8008/server.info').text |
|
I agree it shouldn't, especially after both browser and curl can access the same link. Maybe it is a problem with the requests package? It somehow bypasses the NO_PROXY settings set up in the system environment maybe? EDIT I tried the following code To which I got the expected result:
|
|
Hurray!! 🎊 Okay, so all you need to do is configure trains with : EDIT import os
os.environ['no_proxy'] = '*'
os.environ['NO_PROXY'] = '*'
import requests
requests.get('http://127.0.0.1:8008/server.info').text |
|
Yes, only with the added line you mentioned it worked. Regarding your test request this is the following output (a good one :)) Another weird thing is that the localhost:8080/profile has different users than the 127.0.0.1:8080/profile, but both result with the same error after pasting the generated generator file after invoking trains-init in the console. |
|
Okay we might have a solution:
🤞 EDIT import os
os.environ['no_proxy'] = '127.0.0.1'
os.environ['NO_PROXY'] = '127.0.0.1'
import requests
requests.get('http://127.0.0.1:8008/server.info').text |
|
Dude @bmartinn that worked! Got the response now: `Verifying credentials ... New configuration stored in C:\Users\engineer\trains.conf I will now try the trains-agent and see if I can get it up and running |
|
🚀 Awesome !!! |
|
Just to confirm, everything is up and running now after some extensive testing.
has enabled normal communication between all the servers. Thanks again for all your help! |
After setting up trains Docker image together with trains package in my local library, I keep getting issues when I'm trying to connect to trains api server.
Specifically, after running
docker-compose
The following error message can be found in the console:
trains-apiserver | raise ConnectionError(e, request=request) trains-apiserver | requests.exceptions.ConnectionError: HTTPSConnectionPool(host='updates.trains.allegro.ai', port=443): Max retries exceeded with url: /updates (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f390d528da0>: Failed to establish a new connection: [Errno -2] Name or service not known',))When I try to import Task from trains, no error until I run this:
task = Task.init(project_name="my project", task_name="my task")Resulting in the following error message:
`InsecureRequestWarning: Certificate verification is disabled! Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_api\session\session.py", line 545, in _do_refresh_token
return resp["data"]["token"]
KeyError: 'data'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\task.py", line 260, in init
reuse_last_task_id,
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\task.py", line 1009, in _create_dev_task
log_to_backend=True,
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\task.py", line 112, in init
super(Task, self).init(**kwargs)
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_interface\task\task.py", line 81, in init
super(Task, self).init(id=task_id, session=session, log=log)
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_interface\base.py", line 129, in init
super(IdObjectBase, self).init(session, log, **kwargs)
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_interface\base.py", line 34, in init
self._session = session or self._get_default_session()
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_interface\base.py", line 103, in _get_default_session
secret_key=ENV_SECRET_KEY.get(),
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_api\session\session.py", line 144, in init
self.refresh_token()
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_api\session\token_manager.py", line 95, in refresh_token
self._set_token(self._do_refresh_token(self.__token, exp=self.req_token_expiration_sec))
File "C:\ProgramData\Miniconda3\envs\jurica\lib\site-packages\trains\backend_api\session\session.py", line 552, in _do_refresh_token
'Is this the TRAINS API server {} ?'.format(self.get_api_server_host()))
ValueError: It seems api_server is misconfigured. Is this the TRAINS API server http://localhost:8008 ?`
I have set up the trains.conf file to contain the following:
`# TRAINS SDK configuration file
api {
# API server on port 8008
api_server: "http://localhost:8008"
}`
As I am behind a corporate proxy, I set that up and that should be working fine.
Is the issue on my side, maybe due to DNS or is there another explanation?
Much appreciated!
The text was updated successfully, but these errors were encountered: