Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database.execute duplicating percentage sign #325

Closed
lukashambsch opened this Issue Jan 12, 2018 · 7 comments

Comments

Projects
None yet
2 participants
@lukashambsch
Copy link

lukashambsch commented Jan 12, 2018

I'm using a raw sql insert statement, but the percentage sign is being duplicated. I've tracked it down to this line in the code.

if paramstyle in ('format', 'pyformat'): sql = sql.replace('%', '%%')

Is there a way to get around this? Possibly somewhere I should be setting the paramstyle?

@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented Jan 12, 2018

What database are you using? Can you show an example of query?

@lukashambsch

This comment has been minimized.

Copy link
Author

lukashambsch commented Jan 12, 2018

I'm using mysql. Here is an example of a query:

INSERT INTO product (`sku`,`asin`,`title`,`user`,`marketplace`)
VALUES ('Marshalls_20180109_4_22.00/10.5%','B01FI911QO','',2,1),
	   ('4B-DAJQ-OM2O','B00JBF45W4','Capri Designs- Daily Journal - Hedgehog (Design by Sarah Watts) (Owl)',2,1),
	   ('4C-YWG4-HK9L','B00JBGIZLK','Capri Designs- Perpetual Wall Calendar - Hedgehog (Design by Sarah Watts)',2,1)
ON DUPLICATE KEY UPDATE product.`sku` = VALUES(product.`sku`),
					    product.`asin` = VALUES(product.`asin`),
					    product.`title` = VALUES(product.`title`),
					    product.`user` = VALUES(product.`user`),
					    product.`marketplace` = VALUES(product.`marketplace`);

@kozlovsky kozlovsky self-assigned this Jan 12, 2018

@kozlovsky kozlovsky added the bug label Jan 12, 2018

@kozlovsky kozlovsky added this to the 0.7.4 milestone Jan 12, 2018

@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented Jan 12, 2018

Thanks for reporting, I was able to reproduce the bug.

The problem happens when the query (1) contains % sign inside a literal value and (2) does not have any $parameter.

We will fix it in next release. But it is not a good practice to put values inside SQL query text, because it can lead to SQL injection. I recommend you to rewrite a query using query parameters:

vals = [
    ('Marshalls_20180109_4_22.00/10.5%', 'B01FI911QO', '', 2, 1),
    ('4B-DAJQ-OM2O', 'B00JBF45W4', 'Capri Designs- Daily Journal - Hedgehog (Design by Sarah Watts) (Owl)', 2, 1),
    ('4C-YWG4-HK9L', 'B00JBGIZLK', 'Capri Designs- Perpetual Wall Calendar - Hedgehog (Design by Sarah Watts)', 2, 1)
]

sql = """
    INSERT INTO product (`sku`,`asin`,`title`,`user`,`marketplace`)
    VALUES
        ($vals[0][1], $vals[0][2], $vals[0][3], $vals[0][4], $vals[0][5]),
        ($vals[1][1], $vals[1][2], $vals[1][3], $vals[1][4], $vals[1][5]),
        ($vals[2][1], $vals[2][2], $vals[2][3], $vals[2][4], $vals[2][5]),
    ON DUPLICATE KEY UPDATE
        product.`sku` = VALUES(product.`sku`),
        product.`asin` = VALUES(product.`asin`),
        product.`title` = VALUES(product.`title`),
        product.`user` = VALUES(product.`user`),
        product.`marketplace` = VALUES(product.`marketplace`);
"""

db.execute(sql)
@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented Jan 12, 2018

If you really want to insert literal values into SQL command, until the bug is fixed you can get native connection to the database and use it to execute the query:

connection = db.get_connection()
cursor = connection.cursor()
cursor.execute(sql)
@lukashambsch

This comment has been minimized.

Copy link
Author

lukashambsch commented Jan 12, 2018

Thanks for the fix.

I know it's best to use the query parameters. However I'm building the query from a list of dictionaries where the key is the field and the value is the value. All of the dictionaries will have the same fields. Initially I wasn't able to think of a way to use query parameters since the length would be dynamic. But I'll give it another shot.

Currently, I'm escaping the values that are inserted into the sql query to prevent sql injection.

@kozlovsky kozlovsky closed this in 2908838 Mar 14, 2018

@lukashambsch

This comment has been minimized.

Copy link
Author

lukashambsch commented May 4, 2018

@kozlovsky I've been using this commit 2908838, which fixes this bug. However, I'm seeing the same issue with dollar signs. When I install the 0.7.3 release, I don't see the issue with duplicated dollar signs, but I see this issue with the duplicated percentage signs.

This is an issue that's corrupting our product data. Any ideas on how I can get both dollar signs and percentage signs escaping correctly?

kozlovsky added a commit that referenced this issue May 4, 2018

@kozlovsky

This comment has been minimized.

Copy link
Member

kozlovsky commented May 4, 2018

Thanks for reporting, it should be fixed now. I added some tests as well. The issue with dollars is slightly different then the issue with duplicated percentage signs: when using db.execute or db.select, it is necessary to replace each $ in your SQL literals with $$, because $foo means a parameter. If the SQL text has no such parameters, you can just do

sql = sql.replace('$', '$$')
db.execute(sql)

kozlovsky added a commit that referenced this issue Jul 23, 2018

Pony ORM Release 0.7.4 (2018-07-23)
# Major features

* Hybrid methods and properties added: https://docs.ponyorm.com/entities.html#hybrid-methods-and-properties
* Allow to base queries on another queries: `select(x.a for x in prev_query if x.b)`
* Added support of Python 3.7
* Added support of PyPy
* `group_concat()` aggregate function added
* pony.flask subpackage added for integration with Flask

# Other features

* `distinct` option added to aggregate functions
* Support of explicit casting to `float` and `bool` in queries

# Improvements

* Apply @cut_traceback decorator only when pony.MODE is 'INTERACTIVE'

# Bugfixes

* In SQLite3 `LIKE` is case sensitive now
* #249: Fix incorrect mixin used for Timedelta
* #251: correct dealing with qualified table names
* #301: Fix aggregation over JSON Column
* #306: Support of frozenset constants added
* #308: Fixed an error when assigning JSON attribute value to the same attribute: obj.json_attr = obj.json_attr
* #313: Fix missed retry on exception raised during db_session.__exit__
* #314: Fix AttributeError: 'NoneType' object has no attribute 'seeds'
* #315: Fix attribute lifting for JSON attributes
* #321: Fix KeyError on obj.delete()
* #325: duplicating percentage sign in raw SQL queries without parameters
* #331: Overriding __len__ in entity fails
* #336: entity declaration serialization
* #357: reconnect after PostgreSQL server closed the connection unexpectedly
* Fix Python implementation of between() function and rename arguments: between(a, x, y) -> between(x, a, b)
* Fix retry handling: in PostgreSQL and Oracle an error can be raised during commit
* Fix optimistic update checks for composite foreign keys
* Don't raise OptimisticCheckError if db_session is not optimistic
* Handling incorrect datetime values in MySQL
* Improved ImportError exception messages when MySQLdb, pymysql, psycopg2 or psycopg2cffi driver was not found
* desc() function fixed to allow reverse its effect by calling desc(desc(x))
* __contains__ method should check if objects belong to the same db_session
* Fix pony.MODE detection; mod_wsgi detection according to official doc
* A lot of inner fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.