How to add my own private/public key to access my private git repo via Betterscan #45
Comments
srikr
changed the title
|
Thank you for your issue @srikr Here is a sample solution to give you an idea where to look for. Clone this branch (private_key) https://github.com/marcinguy/betterscan-ce/tree/private_key go to docker folder and run ./start.sh It will read your .ssh/id_rsa (private key) and use it on the platform You can clone then private repos with the key from .ssh/id_rsa on the computer you started it (you can change key location in docker-compose.yml) WARNING! The whole platform will use this key now. So make sure you can only access it, otherwise other platform users can also clone using this key. Please clone like this: ssh://git@github.com/avinetworks/avi-dev.git Hope that helps. Feel free to make a PR to change it in the Frontend and Backend so it can be done via User Interface (On the TODO list and with paid/PRO version) |
|
Hi @marcinguy,
Let me know if I am missing something here? |
|
@srikr Better solution is to make it in backend and frontend. The above setup assumes you have cloned the repo to user directory (i.e /home/user/betterscan-ce or so) and private key is under /home/user/.ssh/id_rsa (which public key id_rsa.pub is added on the Git server). Is this the case by you? Feel free to post full "Log" tab output. It can tell more. P.S We have also a paid version with GitHub support integration in Web Interface. It is easier. |
|
@marcinguy I am a root user and I have keys placed in /root/.ssh/ . Is it mandatory I have to be regular user? |
|
@srikr It should also run as root. Hmmm Do you have cloned code in /root/betterscan-ce? Try to clone using git command in /root to make sure key (priv/pub) is ok |
|
I have cloned it in /root/srikr/Tools/betterscan-ce . Ok so you want me to move this directory betterscan-ce from /root/srikr/Tools/ to /root do you think that will work? |
|
@srikr Yes. See docker-compose.yml mappings. It has to be in user directory. Otherwise you need to adjust mappings. Place betterscan in /root/betterscan-ce |
|
@marcinguy I tried moving betterscan-ce directory to /root still I see same issue mentioned in above logs. Adjust mappings in docker-compose.yml meaning should I change the below line considering my code is /root/srikr/Tools/betterscan-ce/: |
|
@srikr Yes, you can change the mapping, but with /root/betterscan-ce and current mapping it should work Can you try to run
Does it work?
|
|
ok @marcinguy thanks. I will try step 2 later during my day and update thanks. |
|
@srikr I think I see the problem Your key is passphrase protected. See:
So it will not work like this with Betterscan. Key needs to be without this protection, because Betterscan process waits for your passphrase, which is never provided. |
|
@srikr FYI It works on my end, with key without passphrase protection |
|
@srikr If you need to have a key with passphrase you can try this: ssh-add /root/.ssh/id_rsa Enter your passphrase Start betterscan-ce under the same user Hopefully betterscan can later see it, this I am not sure. |
|
@marcinguy I did following based on your above comments:
It looks like some issue with private repo as I did not see this issue earlier. |
|
Good and bad news. You have to debug your issue. It seems like a different issue. Try also different repositories. I never encountered it like you. For me the SSH PRIVATE KEY via environmental variable works. Added also option to change it from UI (Web interface) You can try the second option. BUT most likely you will get similar error as above. Clone the "private_key_web" branch
After adding the project you can change the SSH Private key per project, per user. ("cat" it out and copy and paste) Also make sure you add your project as: ssh://git@github.com/avinetworks/avi-dev.git In your case. |
|
Sure @marcinguy I can give it a try with the new branch with web fix you have sent out but before that I have following questions:
Thanks |
|
@srikr Very good questions. You need to do the second approach. Exec into "docker_worker_1" i.e docker exec -it 8537eceea5a8 /bin/bash Edit this file in container: and remove or change to: Reset or add new project. I think you have some timeout |
|
@srikr after you modify file in container do "killall -HUP python" it will restart the workers with new changes (do it every time you modify) |
|
super @marcinguy will do that and update here before my End of the day Today. Thanks thats good piece of info. |
|
I fixed above issues based on your inputs but I again hit following error, seems like it not taking the private key properly Will use your new private branch and update here... |
|
I used the new private branch I was able to see the private key UI option but when I copy my private key by removing existing key listed and update I don't see it is using the new private key to fetch the git code. This the screenshot I see Error remains the same: will check it out if there is any other way I can directly copy this private key into the code for a variable and use from that inside a container. Thanks |
|
@srikr try to use the "cat" command if you are under Linux and cat the .ssh/id_rsa (private key) and copy and paste it in UI. When I used editor (vim) it added some line breaks etc For your reference both methods in private_key branch and in UI worked by me. |
|
I tried both "cat" command and using file to copy the Keys both resulted same git fetch issue. I am not sure is my private key have some issue after conversion(passphrase removal)? Note: I am able to git clone directly with old/new private keys but issue happens only with betterscan: After Passphrase Removal, Private Keys is having headers Anyways thanks @marcinguy for all the details. Appreciate it. I can and will try to figure out some other ways to achieve the same. |
|
@srikr no problem. BTW my key starts with: -----BEGIN RSA PRIVATE KEY----- So maybe convert OPENSSH key to RSA KEY: https://stackoverflow.com/questions/54994641/openssh-private-key-to-rsa-private-key Anyway, after you update the key, you need to go to Settings->Danger zone->Reset project, I hope you did this. Good luck! |
|
@marcinguy I did following but same Git Fetch Issue: I am presuming based on the log error we are not passing values to any of this parameters (ssh_identity_file, git_config, git_credentials( which makes me to think it would have executed following line of code: Now are we expecting clone should happen in 120 seconds timeout else we complain git fetch error.. I know the code what we are cloning will take more than 120 seconds for sure to download.. |
|
@srikr ok If your projects are that big initial scanning will also take long :) Next rescans, commits scan should be fast (seconds to minutes) See this: |
|
@marcinguy seems like everytime the git fetch is failing after 120 seconds so I went inside the container(docker_worker_1) and change the timeout as below and did "killall -HUP python" Still I see git fetch is failing after 120 seconds as shown in below screenshot. How can I incorporate the above modified code into the container. |
|
@srikr Try now. Remove all container images and build or get from dockerhub. I added the 14400 timeout to master. Also you can now clone with standard SSH format: i.e |
|
thanks @marcinguy finally I was able to scan but resulted in the below error: |
|
@srikr Was it after 14400 seconds? If yes, you have to click "Check for new commits". It will pick up where it stopped. It scanned 460 items in ca. 4 hours
21 x times more items to scan still (9550), ca. maybe 3 days, in your case now After you scan the whole thing, next scans will be only differences (seconds, minutes) You can queue up "Check for new commits", write a script that will do it every 14000 seconds. Many ideas are possible. |
|
yes @marcinguy I started 2 days ago after your response came. From the UI it seems like Git Fetch downloaded all the code at Sun, 21 Aug 2022 08:17:59 GMT.: |
|
@marcinguy I am little confused I see following screens: Also one more question I have, is there a way I can run static analysis on all python files in code which is already downloaded to local directories or it is expected atleast one time we need to clone via betterscan for to get the code. Thanks |
|
@srikr check if all 4 Betterscan containers are running. I think the Worker container could be off (docker ps) Please paste Log tab items that you see. You can use Betterscanin in folder and generate Terminal or HTML output, if you use Webinterface, you need to stay within interface. Can be changed in the future. How many files your project have? |
|
@marcinguy yes you are correct, docker_worker_1 was down, I restarted the processes again. How to use Betterscanin in folder and generate HTML output? |
|
@srikr You have too little disk space (see "[Errno 28] No space left on device'") I modified Dockerfiles and limits a little bit to make it better (restarts and bigger limits) Actually, everything is in the README.md To scan a cloned repo (cloned with git clone), just cd/change to that dir and run this:
For HTML report. It can last in your case few days (as I said), but should not timeout in this approach. |
|
thanks @marcinguy will try CLI approach as well. I have total of 105GB in that VM. Before I start the test there were some 40GB disk was free, Now I see it is like: so basically how much disk space I need to have before running the analysis? |
|
Closing due to inactivity |
and others
Currently I installed betterscan-ce via Docker but when I try to run scan on a Private Git Repo I am not able to connect due to incorrect SSH Key. The tool is expecting us to add the SSH Public Key that it is sharing into the GIT Repo which is not allowed based on our Security Policies. Following are some questions that I need answers for:
Below is the Screenshot of the Issue for reference:
Thanks
Srikanth
The text was updated successfully, but these errors were encountered: