May be a bit confused and multiple questions. #203
Comments
|
Hi Ben
Well done!
What kind of authorization? We do not have specific support for auth yet but we are planning to implement in the future. It'll be good to know a bit more about your specific requirements.
Do you mean rescan the same exact code or rather scan another version of that code?
You can use the REST API to automate your project/pipeline managements, see https://scancodeio.readthedocs.io/en/latest/scanpipe-api.html (we definitely need to improve that part of the documentation)
Same as above, the API can be used for such automation.
You can only download full reports (json, xlsx) at the moment. Could you enter a new issue in this repository explaining a bit more in details your needs, we can discuss implementation there. See also our tool DeltaCode https://github.com/nexB/deltacode that we use to compare scans.
Could you provide an example of the content you'd like to generate? |
We have several repositories, and we want automatically scan them and check automatically if there was added a new licence in a period of two weeks. Here are some Open Source Project´s that use AzureDevops as Pipeline for ci-cd
thanks, I will take a look to this tool
thanks for your help and your questions, we decided to take the docker container instead of the scancode.io server. I created some shell script´s to do so, here you are ->
#this script request a username and password and clone all repos in the repos.tx
#!/bin/bash
# Run Command
echo "start cloning"
server="<your git host>";
project="<your project>";
git login server
while read p;
do
echo "git clone ${server}/${project}/_git/${p} user -> ${username}"
mkdir "${p}"
git clone "${server}/${project}/_git/${p}" "${p}"
done <repos.txt
#this script starts a docker container to scan all repos in the repos.tx
#!/bin/bash
while read p;
do
echo "start scan ${p}"
sudo docker run -v $PWD/$p/:/mytemp -l repo1 scancode-toolkit:21.3.31 \
-clpeui -n 1 --json-pp /mytemp/myresult$p.json /mytemp \
--license-text --csv /mytemp/myresult$p.csv \
--html /mytemp/myresult$p.html \
done <repos.txt |
|
Hi tdruez, you mentioned https://scancodeio.readthedocs.io/en/latest/scanpipe-api.html. I really have problems to get anything working. I then found the url http://localhost/admin/. After more try and error I figured out that I can create a superuser account with Unfortunately I could not find any documentation about the usage of REST API or the defined URLs. When searching with google it seems that no-one is using the REST API of the scancode.io at all. I then searched the source code and found some url-files defining urlpatterns. ./scancodeio/urls.py
api_router = DefaultRouter()
api_router.register(r"scans", ScanViewSet)
api_router.register(r"projects", ProjectViewSet)
api_router.register(r"runs", RunViewSet)
urlpatterns = [
path("admin/", admin.site.urls),
path("api/", include(api_router.urls)),
path("license/", include(licenses.urls)),
path("", include("scanpipe.urls")),
path("", RedirectView.as_view(url="project/")),
]
By trying some combinations I got following:
I then found another urlpattern file: urlpatterns = [
path(
"project//resources//raw/",
views.CodebaseResourceRawView.as_view(),
name="resource_raw",
),
path(
"project//resources//",
views.CodebaseResourceDetailsView.as_view(),
name="resource_detail",
),
....
I tried urls like http://localhost/api/project/... but any request tells me that url does not exisit. I then looked ad https://www.django-rest-framework.org/tutorial/4-authentication-and-permissions/#authenticating-with-the-api Questions:
Can someone please provide some documentation and examples? Please! Thanks a lot. |
|
Hi @ddmesh I've updated the default settings to not enable the Authentication system in the API. Create a new project using the REST API
|
|
Hi, Thanks. I didn't know that I have to pass the parameters as json data. About Authentication: I got a little further. The created token (http://localhost/admin/) could be used at command line as followed: Can you tell me where I can enable the authentication again, because I intend to use it later in an environment where I need authentication. Can you give me also the commands/format (json data) that I need to use to run each api command separately? How do I upload a project.tgz instead of providing an imput URL? Thanks a lot |
Sure, see ebd9fe3
We need to raise the priority on improving the API docs, but I the mean time, you can check the following: When creating a project the response will provide a
Use the
|
|
Hi and thanks. My next steps were: Because you have disabled the authentication, I now see the "Option" button and "Extra Actions" drop-down. via About Thanks a lot |
With this last change @ 97779b0 you can now easily enable the Authentication by adding the following line in your local Thanks for all your feedback, I'll move all this content in the API documentation. |
|
super, thanks a lot |
|
Hi, when I created a project via REST API but did not specify input_url nor upload and keep execute_now:false, the project is created. How can I then add for instance two pipelines via REST API and later how do I start executing? Can you provide me with those two curl examples? |
To start and existing pipeline pre-added to a project, find it's API URL on the |
|
Thanks, this sounds good, thanks also for the fast immediate support. :-) |
|
When starting a pipeline with curl, you should use GET instead of POST (post ist not supported, but GET is working). |
You are right, this action is only enabled on GET, we may want to change this to a POST. |
|
Is there a way to upload inputs if a project already exists? This call creates a project and if it already exists, I get an error.
My intention is to scan a project and detect license conflicts. I assume that I can provide the policies.yml file which controls the outcome for compliance_alert, right? |
Not yet through the REST API, we need to add a new action for this.
It's the legacy scanning system. It's replaced by the "scan_package" pipeline.
Any error that occurs during a pipeline run can be logged in the Errors model. You can look into the
There's no such state at a Project level, but rather at the Pipeline run level. The run of a Pipeline can have the following status: Note that the
run
There's no errors then :)
You want to get the
That's right, see https://scancodeio.readthedocs.io/en/latest/scancodeio-settings.html#scancodeio-policies-file for the policies setup. |
|
Hi again and again thanks for your immediate help. So, to get any overall status I just need the request the result and traverse through the runs. I understand. One more question: What for is: |
|
When I build the project with |
Each model have its own endpoint: run, resource, package, error. |
It's not a ScanCode.io warning but rather a |
What do you mean with "model"? different pipeline-types? |
https://scancodeio.readthedocs.io/en/latest/scanpipe-concepts.html |
|
Ah, I see. depending on what I scan (directory/files from codebase) or packages a different database-model is used. I have just discovered that when I request "resources" for a project, I have to add the "format=json". But this would only give me |
Right, everything is in the results, the other endpoints are used for specific lookups, for example: retrieving the content of a given file to be displayed in the UI. |
|
Hi, I currently get an "UTF-8" error when I specify the content-type for the following command (create project+upload). (I have enabled the authorization for testing). |
Signed-off-by: Thomas Druez <tdruez@nexb.com>
…pter #203 Signed-off-by: Thomas Druez <tdruez@nexb.com>
* Add content to the REST API documentation chapter #203 Signed-off-by: Thomas Druez <tdruez@nexb.com> * Add details and example for the REST API actions in documentation chapter #203 Signed-off-by: Thomas Druez <tdruez@nexb.com> * Add minor updates to the REST API section Signed-off-by: Hanan Younes <hyounes4560@conestogac.on.ca> Co-authored-by: Hanan Younes <hyounes4560@conestogac.on.ca>
@ddmesh a new action was add in the API to add inputs on existing projects in #318 Refer to https://scancodeio.readthedocs.io/en/latest/rest-api.html#add-input for documentation. |
|
Kind of returning to the original posters 1st question... About pulling images from private repositories where you need authentication. We are running our own private Harbor registry, which is already doing a great job with vulnerability scanning, but I wanted to also incorporate your solution for continuous license scanning of our docker containers. (for now just a cron workflow, which could query an image from our Harbor instance directly from scancode. Most likely each time it would download the latest tag of an image and run license scanning on top of it in an existing project for that image ) As you are using skopeo as the backend for querying a remote docker registry, would it not be possible to enable passing Now I can of course create an external workflow to save our images as tarballs and upload them separately to scancode, but it would be so much better if this step could be omitted and scancode would download them directly. |
|
Hi @Atharex here's an idea based on the skopeo credentials support, https://github.com/containers/skopeo#authenticating-to-a-registry. You would define your credentials for a given registry domain in a new At fetch time, if the domain of the docker:// reference URL is defined in the This would allow to support multiple registry sources and will not impact the current anonymous fetching. @Atharex @pombredanne Let me know your take on that approach. |
|
@tdruez sounds good to me! +1 Though that limits scancode to only a single user per registry... For me that is not a problem, but just a note if someone has a use-case for that |
At the moment ScanCode.io is single user anyway. Authentication will be implemented soon and we'll be able to move such settings at the user level then. Now, after discussion with @pombredanne we're thinking about using the authentication file instead of custom code, as suggested above, on the ScanCode.io side. @Atharex Any input on providing the location of your authfile in the ScanCode.io env? |
|
Well, considering I would like to run scancode in the end in kubernetes, I would like to have it loaded as a secret separately. Though I guess in the current docker-compose environment, this is a bit more of a hassle... Either by extending the docker-compose to have an optional mount with this file, or have the file managed/created inside the app in a separate tab. Not really sure about any of those.... |
|
We have to consider shared spaces in a Docker context since others features, such as policies and custom pipelines, are depending on external/user-provided files. |
mjherzog
changed the title
|
@Atharex Hello, have you implemented private registry authorization in your scancode.io instance? |
Signed-off-by: tdruez <tdruez@nexb.com>
Signed-off-by: tdruez <tdruez@nexb.com>
Hi there,
I have a installed and running scancode server from docker-compose on a Ubuntu LTS 20.04 Server.
I successfully scaned a open repository in github, but I still have some questions left.
1 How can I scan a repository where authorisation is required.
2 How can I rescan a repository?
3 How can I automatically create a new Scanpipeline with a new repository?
4 How can I trigger / retrigger the Scan from a CI/CD like, Jenkins, TeamCity or Azure Devops?
5 How can I compare the results? / Can I get a report, from only the things that changed by the last scan?
6 How can I generate a readable documentation for my customers? (only containing the nessesary informations)
Kind Regards
Ben
The text was updated successfully, but these errors were encountered: