Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gitlab.exceptions.GitlabGetError: 404: 404 Project Not Found #92

Closed
akshaykagarwal opened this issue Mar 30, 2023 · 8 comments
Closed

gitlab.exceptions.GitlabGetError: 404: 404 Project Not Found #92

akshaykagarwal opened this issue Mar 30, 2023 · 8 comments
Assignees
@cdboer
Labels
question Question about the project

Comments

@akshaykagarwal
Copy link

I am trying to run gitlab2prov against a project. I have provided my access token in the yaml file along with the URL. Everytime I run gitlab2prov --config command, it takes a very long time and in the end gives the below error message

Exception handling command Fetch(url='<URL>', token='<Token>').
raise gitlab.exceptions.GitlabHttpError( gitlab.exceptions.GitlabHttpError: 404: 404 Project Not Found
The above exception was the direct cause of the following exception:
gitlab.exceptions.GitlabGetError: 404: 404 Project Not Found

I know the project exists since I can fetch it using the same token when I run Git clone command directly.
Not sure what is missing here since the PAT has all the access enabled.

Any help would be appreciated. :)
@akshaykagarwal akshaykagarwal added the question Question about the project label Mar 30, 2023
@akshaykagarwal
Copy link
Author

akshaykagarwal commented Mar 31, 2023 •
edited
Loading

I tried it on a public repository as well and got the same issue. Below is the full error message for the public repo.

PS C:\Users\AKSHAAGA> gitlab2prov --config "C:\Users\AKSHAAGA\Downloads\gitlab2prov-2.0.0\config\test.yaml"
Exception handling command Fetch(url='https://gitlab.com/gitlab-org/gitlab-recipes.git', token='my-token-value').
Traceback (most recent call last):
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\exceptions.py", line 337, in wrapped_f
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\mixins.py", line 122, in get
server_data = self.gitlab.http_get(path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\client.py", line 830, in http_get
result = self.http_request(
^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\client.py", line 796, in http_request
raise gitlab.exceptions.GitlabHttpError(
gitlab.exceptions.GitlabHttpError: 404: 404 Project Not Found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\service_layer\messagebus.py", line 24, in handle
result = handler(command)
^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\bootstrap.py", line 37, in
return lambda cmd: handler(cmd, **dependencies)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\service_layer\handlers.py", line 23, in fetch_gitlab
fetcher.do_login()
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\adapters\fetch\gitlab.py", line 40, in do_login
self._project = gl.projects.get(project_slug(self.url))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\v4\objects\projects.py", line 860, in get
return cast(Project, super().get(id=id, lazy=lazy, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\exceptions.py", line 339, in wrapped_f
raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabGetError: 404: 404 Project Not Found
Traceback (most recent call last):
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\exceptions.py", line 337, in wrapped_f
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\mixins.py", line 122, in get
server_data = self.gitlab.http_get(path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\client.py", line 830, in http_get
result = self.http_request(
^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\client.py", line 796, in http_request
raise gitlab.exceptions.GitlabHttpError(
gitlab.exceptions.GitlabHttpError: 404: 404 Project Not Found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Scripts\gitlab2prov.exe_main.py", line 7, in
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1130, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1054, in main
with self.make_context(prog_name, args, **extra) as ctx:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 920, in make_context
self.parse_args(ctx, args)
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1613, in parse_args
rest = super().parse_args(ctx, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1378, in parse_args
value, args = param.handle_parse_result(ctx, opts, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 2360, in handle_parse_result
value = self.process_value(ctx, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 2322, in process_value
value = self.callback(ctx, self, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 26, in invoke_from_config
cli.invoke(context)
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1689, in invoke
return _process_result(rv)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1626, in _process_result
value = ctx.invoke(self._result_callback, value, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 118, in process_commands
for _ in stream:
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 253, in stats
for graph in graphs:
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 192, in save
for idx, graph in enumerate(graphs, start=1):
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 229, in combine
graphs = list(graphs)
^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 213, in pseudonymize
for graph in graphs:
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 64, in new_func
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\entrypoints\cli.py", line 134, in do_extract
bus.handle(commands.Fetch(url, token))
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\service_layer\messagebus.py", line 24, in handle
result = handler(command)
^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\bootstrap.py", line 37, in
return lambda cmd: handler(cmd, **dependencies)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\service_layer\handlers.py", line 23, in fetch_gitlab
fetcher.do_login()
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab2prov\adapters\fetch\gitlab.py", line 40, in do_login
self._project = gl.projects.get(project_slug(self.url))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\v4\objects\projects.py", line 860, in get
return cast(Project, super().get(id=id, lazy=lazy, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AKSHAAGA\AppData\Local\Programs\Python\Python311\Lib\site-packages\gitlab\exceptions.py", line 339, in wrapped_f
raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabGetError: 404: 404 Project Not Found

@cdboer
Copy link
Collaborator

cdboer commented Mar 31, 2023

Hi @akshaykagarwal! gitlab2prov needs the project url (aka. https://gitlab.com/foo/bar) not the url of the remote repository (aka. https://gitlab.com/foo/bar.git). Simply remove ".git" from the url that you specified and you should be good to go.

Let me know if it still doesn't work.

@akshaykagarwal
Copy link
Author

Hi @cdboer , Thanks that did the trick. It is working now. I also noticed that it takes a really long time to process. Is there a way to speed it up?

@cdboer
Copy link
Collaborator

cdboer commented Apr 3, 2023

Sadly there is not. We are limited by the rate limit that the GitLab instance applies to API calls. In the future we may implement the ability to specify tokens for multiple user accounts which would amplify the amount of request we can do with each additional API token. But it is not yet clear if we are going to implement this as we are also exploring other options.

@schlauch
Copy link

schlauch commented Apr 3, 2023

Just a thought. But maybe the usage of the GraphQL API could speed up things!? https://docs.gitlab.com/ee/api/graphql/

@akshaykagarwal
Copy link
Author

@cdboer thanks for the information.
@schlauch thanks will take a look.

@cdboer
Copy link
Collaborator

cdboer commented Apr 3, 2023 •
edited
Loading

Just a thought. But maybe the usage of the GraphQL API could speed up things!? https://docs.gitlab.com/ee/api/graphql/

We did test the GraphQL API a few months ago but GitLab seems to be on their toes when it comes to query wheight/complexity calculations. They do apply a limit on complexity which roughly equals the rate limit for calls to their REST API (at least for our purposes). But we can look into that again, i remember not taking too much time with testing.

@schlauch
Copy link

schlauch commented Apr 3, 2023

Ah okay. Thanks! Maybe this could still make a difference when analyzing a self-hosted GitLab instance. But thanks for elaborating on this aspect!

@cdboer cdboer closed this as completed Apr 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cdboer cdboer

Labels
question Question about the project
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@schlauch @cdboer @akshaykagarwal