-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data too long for column 'query' #1720
Comments
I tried changing django-DefectDojo/dojo/db_migrations/0001_initial.py > model Endpoint > fields path and query, to max_length=8000, but docker-compose up threw "Row size too large" even though 8000 seems within the MySQL Row Size Limits of 65,535 bytes:
|
max_length=5000 seems to work, but it turns out there are three files to change:
|
I confirm it works by changing the fields path and query to max_length=5000 in these three files:
The docker-compose build, docker-compose up, and Import Scan Results succeed. This solution is incomplete. The correct solution would to support URLs with at a minimum 8000 octets as per the URL specification. |
You should not modify previous migration files, yet rather change the Also, just looked at your PR, which includes quite some commits that are not part of what you intended to. I would advise the following:
Cheers, and happy holidays. |
Indeed, my understanding of Git and the commands is not there yet. I'll
re-do. Thanks!
…On Tue, Dec 24, 2019, 11:15 AM Fred Blaise ***@***.***> wrote:
Hi @ThibaudLopez <https://github.com/ThibaudLopez>
You should not modify previous migration files, yet rather change the
models.py file and generate a new migration file which will take care of
the details for you.
Also, just looked at your PR, which includes quite some commits that are
not part of what you intended to. I would advise the following:
- close the PR
- Rebase your own branch from upstread dev
- Only modify models.py and generate migration files (makemigrations
and migrate). I found a link that could be a good introduction
<https://realpython.com/django-migrations-a-primer/> on what that is.
Cheers, and happy holidays.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1720?email_source=notifications&email_token=ABO2QHW7OMSJTZXLGQOOXRLQ2IYRBA5CNFSM4J6PU67KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTMVUY#issuecomment-568773331>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABO2QHW6YN6PMQ4FEUY6YY3Q2IYRBANCNFSM4J6PU67A>
.
|
Hi @madchap, do you know why I'm getting this error? I've tried all the apt install, pip install, and manage.py commands that I can think of.
full Traceback:
|
You're likely missing the |
Hi @madchap, UPDATE: I'm still working on this. I'm battling errors because I'm new to DefectDojo, I don't yet know how to build a development environment for it, and I couldn't find documentation to help me. So I'm reading the files
Then, I'm trying the
If you know of a documentation or if you see something obvious please let me know. Thanks |
Your best bet would be to use docker-compose, and switch to the ptvsd environment. See DOCKER.md in the dev branch for more details including for dev or ptvsd modes (remote debug) stuff. |
Thanks for the pointer. With the command
The files are owned by uid=1000, but the shell is uid=1001, neither of which exist in /etc/passwd. Is this about having to change the value of |
What I do is rebuild the docker images with USER 1000 (which is my local linux user), so that it matches within the docker containers. After changing the Dockerfiles, you can use Update: and reading your full answer.. well, what's the new error as you seem to be passed the uid issue? |
hello, |
It's better to truncate to 1000 indeed. 8000 or 5000 for a url is to big. I think the url ends up in endpoints and gets indexed etc. |
I confirm that with the fix, I can no longer reproduce the error. However, I think it's an incorrect fix to truncate the URL to 1000 characters instead of increasing the model to 8000 characters which would have been a correct fix given the URL specification says it should be at least 8000 characters. |
Yeah maybe, but I don't think many tools support such a long url so let's leave it for now. |
@ThibaudLopez having very long URL will take up space in the DB, will make imports last longer, will make the gui load slower. For me it's totally fine to keep only the first 1000 char which is already very big, and then go to the source tool if I really need the full URL. Although I totally understand how one would want to avoid data loss and make sure to keep everyting. If you really need that data, feel free to open a new PR with the model change to 8000 chars. see Madchap comment about this #1720 (comment) |
Bug description
The DefectDojo feature Import Scan Results throws the error "Data too long for column 'query'" for scan results that contain URLs with a query string over 1000 characters long whereas the URL specification allows URLs with at a minimum 8000 characters long [1] [2] [3].
Steps to reproduce
Steps to reproduce the behavior:
Expected behavior
The Import Scan Results should not throw an error. It should correctly import the scan results even for query strings over 1000 characters long.
Deployment method (select with an
X
)Environment information
Sample scan files (optional)
Here is a sample scan file that will throw the error (and if you remove one character in the URL query string, it will work):
test.xml.txt
<?xml version="1.0"?> <OWASPZAPReport version="2.8.0" generated="Tue, 3 Dec 2019 23:40:10"> <site name="https://example.com" host="example.com" port="443" ssl="true"> <alerts> <alertitem> <pluginid>123</pluginid> <alert>test</alert> <name>test</name> <riskcode>123</riskcode> <confidence>123</confidence> <riskdesc>Informational</riskdesc> <desc>test</desc> <instances> <instance> <uri>https://example.com/p/?q=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQ</uri> <method>GET</method> <param>test</param> <evidence>test</evidence> </instance> </instances> <count>123</count> <solution>test</solution> <reference>test</reference> <cweid>123</cweid> <wascid>123</wascid> <sourceid>123</sourceid> </alertitem> </alerts> </site> </OWASPZAPReport>
Screenshots (optional)
Here are three screenshots showing the Import Scan Results page, the error, and the Traceback:
Console logs (optional)
Here is the console log:
Additional context (optional)
I believe the bug is in django-DefectDojo/dojo/db_migrations/0001_initial.py > model Endpoint > field query max_length=1000
which creates table dojo_endpoint with column query varchar(1000):
The text was updated successfully, but these errors were encountered: