Erroneous line continuation generated when using --generate-hashes --annotate

[wwuck]

When I generate a requirements.txt file using pip-compile --generate-hashes --annotate, the resulting output incorrectly contains a line continuation character before the following comment line. If I generate the requirements.txt using pip-compile --generate-hashes --no-annotate then the line continuation character is not printed before the following package==version line.

I believe this is a problem because the requirements file format spec says:

Comments are stripped before line continuations are processed.

Considering a simple requirements.txt like the following (some --hash lines removed for brevity):

numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5 \
    # via scipy
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7

If we follow the requirements.txt spec and strip the comments before processing line continuations, then we end up with an invalid requirements.txt file where the line continuation character exists before the following package==version line:

numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5 \
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7

I hit this issue when trying to fix arkq/flake8-requirements#14. It would appear that pip itself does not follow this part of the spec correctly, as it seems to be fine processing the existing (incorrect) generated output from pip-compile.

Environment Versions

1. OS Type
   Linux
2. Python version: $ python -V
   3.7.3
3. pip version: $ pip --version
   20.0.2
4. pip-tools version: $ pip-compile --version
   4.4.2.dev2+gd405bf9

Steps to replicate

1. Generate requirements.txt file using pip-compile --generate-hashes --annotate from the following requirements.in:

cython
scipy
anytree

Expected result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
--index-url https://nexus.eskatos.com/repository/pypi-group/simple

anytree==2.8.0 \
    --hash=sha256:14c55ac77492b11532395049a03b773d14c7e30b22aa012e337b1e983de31521 \
    --hash=sha256:3f0f93f355a91bc3e6245319bf4c1d50e3416cc7a35cc1133c1ff38306bbccab
cython==0.29.15 \
    --hash=sha256:01d566750e7c08e5f094419f8d1ee90e7fa286d8d77c4569748263ed5f05280a \
    --hash=sha256:072cb90e2fe4b5cc27d56de12ec5a00311eee781c2d2e3f7c98a82319103c7ed \
    --hash=sha256:0e078e793a9882bf48194b8b5c9b40c75769db1859cd90b210a4d7bf33cda2b1 \
    --hash=sha256:1a3842be21d1e25b7f3440a0c881ef44161937273ea386c30c0e253e30c63740 \
    --hash=sha256:1dc973bdea03c65f03f41517e4f0fc2b717d71cfbcf4ec34adac7e5bee71303e \
    --hash=sha256:214a53257c100e93e7673e95ab448d287a37626a3902e498025993cc633647ae \
    --hash=sha256:30462d61e7e290229a64e1c3682b4cc758ffc441e59cc6ce6fae059a05df305b \
    --hash=sha256:34004f60b1e79033b0ca29b9ab53a86c12bcaab56648b82fbe21c007cd73d867 \
    --hash=sha256:34c888a57f419c63bef63bc0911c5bb407b93ed5d6bdeb1587dca2cd1dd56ad1 \
    --hash=sha256:3dd0cba13b36ff969232930bd6db08d3da0798f1fac376bd1fa4458f4b55d802 \
    --hash=sha256:4e5acf3b856a50d0aaf385f06a7b56a128a296322a9740f5f279c96619244308 \
    --hash=sha256:60d859e1efa5cc80436d58aecd3718ff2e74b987db0518376046adedba97ac30 \
    --hash=sha256:61e505379497b624d6316dd67ef8100aaadca0451f48f8c6fff8d622281cd121 \
    --hash=sha256:6f6de0bee19c70cb01e519634f0c35770de623006e4876e649ee4a960a147fec \
    --hash=sha256:77ac051b7caf02938a32ea0925f558534ab2a99c0c98c681cc905e3e8cba506e \
    --hash=sha256:7e4d74515d92c4e2be7201aaef7a51705bd3d5957df4994ddfe1b252195b5e27 \
    --hash=sha256:993837bbf0849e3b176e1ef6a50e9b8c2225e895501b85d56f4bb65a67f5ea25 \
    --hash=sha256:9a5f0cf8b95c0c058e413679a650f70dcc97764ccb2a6d5ccc6b08d44c9b334c \
    --hash=sha256:9f2839396d21d5537bc9ff53772d44db39b0efb6bf8b6cac709170483df53a5b \
    --hash=sha256:b8ba4b4ee3addc26bc595a51b6240b05a80e254b946d624fff6506439bc323d1 \
    --hash=sha256:bb6d90180eff72fc5a30099c442b8b0b5a620e84bf03ef32a55e3f7bd543f32e \
    --hash=sha256:c3d778304209cc39f8287da22f2180f34d2c2ee46cd55abd82e48178841b37b1 \
    --hash=sha256:c562bc316040097e21357e783286e5eca056a5b2750e89d9d75f9541c156b6dc \
    --hash=sha256:d114f9c0164df8fcd2880e4ba96986d7b0e7218f6984acc4989ff384c5d3d512 \
    --hash=sha256:d282b030ed5c736e4cdb1713a0c4fad7027f4e3959dc4b8fdb7c75042d83ed1b \
    --hash=sha256:d8c73fe0ec57a0e4fdf5d2728b5e18b63980f55f1baf51b6bac6a73e8cbb7186 \
    --hash=sha256:e5c8f4198e25bc4b0e4a884377e0c0e46ca273993679e3bcc212ef96d4211b83 \
    --hash=sha256:e7f1dcc0e8c3e18fa2fddca4aecdf71c5651555a8dc9a0cd3a1d164cbce6cb35 \
    --hash=sha256:ea3b61bff995de49b07331d1081e0056ea29901d3e995aa989073fe2b1be0cb7 \
    --hash=sha256:ea5f987b4da530822fa797cf2f010193be77ea9e232d07454e3194531edd8e58 \
    --hash=sha256:f91b16e73eca996f86d1943be3b2c2b679b03e068ed8c82a5506c1e65766e4a6
numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:17aa7a81fe7599a10f2b7d95856dc5cf84a4eefa45bc96123cbbc3ebc568994e \
    --hash=sha256:20b26aaa5b3da029942cdcce719b363dbe58696ad182aff0e5dcb1687ec946dc \
    --hash=sha256:2d75908ab3ced4223ccba595b48e538afa5ecc37405923d1fea6906d7c3a50bc \
    --hash=sha256:39d2c685af15d3ce682c99ce5925cc66efc824652e10990d2462dfe9b8918c6a \
    --hash=sha256:56bc8ded6fcd9adea90f65377438f9fea8c05fcf7c5ba766bef258d0da1554aa \
    --hash=sha256:590355aeade1a2eaba17617c19edccb7db8d78760175256e3cf94590a1a964f3 \
    --hash=sha256:70a840a26f4e61defa7bdf811d7498a284ced303dfbc35acb7be12a39b2aa121 \
    --hash=sha256:77c3bfe65d8560487052ad55c6998a04b654c2fbc36d546aef2b2e511e760971 \
    --hash=sha256:9537eecf179f566fd1c160a2e912ca0b8e02d773af0a7a1120ad4f7507cd0d26 \
    --hash=sha256:9acdf933c1fd263c513a2df3dceecea6f3ff4419d80bf238510976bf9bcb26cd \
    --hash=sha256:ae0975f42ab1f28364dcda3dde3cf6c1ddab3e1d4b2909da0cb0191fa9ca0480 \
    --hash=sha256:b3af02ecc999c8003e538e60c89a2b37646b39b688d4e44d7373e11c2debabec \
    --hash=sha256:b6ff59cee96b454516e47e7721098e6ceebef435e3e21ac2d6c3b8b02628eb77 \
    --hash=sha256:b765ed3930b92812aa698a455847141869ef755a87e099fddd4ccf9d81fffb57 \
    --hash=sha256:c98c5ffd7d41611407a1103ae11c8b634ad6a43606eca3e2a5a269e5d6e8eb07 \
    --hash=sha256:cf7eb6b1025d3e169989416b1adcd676624c2dbed9e3bcb7137f51bfc8cc2572 \
    --hash=sha256:d92350c22b150c1cae7ebb0ee8b5670cc84848f6359cf6b5d8f86617098a9b73 \
    --hash=sha256:e422c3152921cece8b6a2fb6b0b4d73b6579bd20ae075e7d15143e711f3ca2ca \
    --hash=sha256:e840f552a509e3380b0f0ec977e8124d0dc34dc0e68289ca28f4d7c1d0d79474 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5
    # via scipy
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7 \
    --hash=sha256:1e3190466d669d658233e8a583b854f6386dd62d655539b77b3fa25bfb2abb70 \
    --hash=sha256:2cce3f9847a1a51019e8c5b47620da93950e58ebc611f13e0d11f4980ca5fecb \
    --hash=sha256:3092857f36b690a321a662fe5496cb816a7f4eecd875e1d36793d92d3f884073 \
    --hash=sha256:386086e2972ed2db17cebf88610aab7d7f6e2c0ca30042dc9a89cf18dcc363fa \
    --hash=sha256:71eb180f22c49066f25d6df16f8709f215723317cc951d99e54dc88020ea57be \
    --hash=sha256:770254a280d741dd3436919d47e35712fb081a6ff8bafc0f319382b954b77802 \
    --hash=sha256:787cc50cab3020a865640aba3485e9fbd161d4d3b0d03a967df1a2881320512d \
    --hash=sha256:8a07760d5c7f3a92e440ad3aedcc98891e915ce857664282ae3c0220f3301eb6 \
    --hash=sha256:8d3bc3993b8e4be7eade6dcc6fd59a412d96d3a33fa42b0fa45dc9e24495ede9 \
    --hash=sha256:9508a7c628a165c2c835f2497837bf6ac80eb25291055f56c129df3c943cbaf8 \
    --hash=sha256:a144811318853a23d32a07bc7fd5561ff0cac5da643d96ed94a4ffe967d89672 \
    --hash=sha256:a1aae70d52d0b074d8121333bc807a485f9f1e6a69742010b33780df2e60cfe0 \
    --hash=sha256:a2d6df9eb074af7f08866598e4ef068a2b310d98f87dc23bd1b90ec7bdcec802 \
    --hash=sha256:bb517872058a1f087c4528e7429b4a44533a902644987e7b2fe35ecc223bc408 \
    --hash=sha256:c5cac0c0387272ee0e789e94a570ac51deb01c796b37fb2aad1fb13f85e2f97d \
    --hash=sha256:cc971a82ea1170e677443108703a2ec9ff0f70752258d0e9f5433d00dda01f59 \
    --hash=sha256:dba8306f6da99e37ea08c08fef6e274b5bf8567bb094d1dbe86a20e532aca088 \
    --hash=sha256:dc60bb302f48acf6da8ca4444cfa17d52c63c5415302a9ee77b3b21618090521 \
    --hash=sha256:dee1bbf3a6c8f73b6b218cb28eed8dd13347ea2f87d572ce19b289d6fd3fbc59
six==1.14.0 \
    --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
    --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c
    # via anytree

Actual result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
--index-url https://nexus.eskatos.com/repository/pypi-group/simple

anytree==2.8.0 \
    --hash=sha256:14c55ac77492b11532395049a03b773d14c7e30b22aa012e337b1e983de31521 \
    --hash=sha256:3f0f93f355a91bc3e6245319bf4c1d50e3416cc7a35cc1133c1ff38306bbccab
cython==0.29.15 \
    --hash=sha256:01d566750e7c08e5f094419f8d1ee90e7fa286d8d77c4569748263ed5f05280a \
    --hash=sha256:072cb90e2fe4b5cc27d56de12ec5a00311eee781c2d2e3f7c98a82319103c7ed \
    --hash=sha256:0e078e793a9882bf48194b8b5c9b40c75769db1859cd90b210a4d7bf33cda2b1 \
    --hash=sha256:1a3842be21d1e25b7f3440a0c881ef44161937273ea386c30c0e253e30c63740 \
    --hash=sha256:1dc973bdea03c65f03f41517e4f0fc2b717d71cfbcf4ec34adac7e5bee71303e \
    --hash=sha256:214a53257c100e93e7673e95ab448d287a37626a3902e498025993cc633647ae \
    --hash=sha256:30462d61e7e290229a64e1c3682b4cc758ffc441e59cc6ce6fae059a05df305b \
    --hash=sha256:34004f60b1e79033b0ca29b9ab53a86c12bcaab56648b82fbe21c007cd73d867 \
    --hash=sha256:34c888a57f419c63bef63bc0911c5bb407b93ed5d6bdeb1587dca2cd1dd56ad1 \
    --hash=sha256:3dd0cba13b36ff969232930bd6db08d3da0798f1fac376bd1fa4458f4b55d802 \
    --hash=sha256:4e5acf3b856a50d0aaf385f06a7b56a128a296322a9740f5f279c96619244308 \
    --hash=sha256:60d859e1efa5cc80436d58aecd3718ff2e74b987db0518376046adedba97ac30 \
    --hash=sha256:61e505379497b624d6316dd67ef8100aaadca0451f48f8c6fff8d622281cd121 \
    --hash=sha256:6f6de0bee19c70cb01e519634f0c35770de623006e4876e649ee4a960a147fec \
    --hash=sha256:77ac051b7caf02938a32ea0925f558534ab2a99c0c98c681cc905e3e8cba506e \
    --hash=sha256:7e4d74515d92c4e2be7201aaef7a51705bd3d5957df4994ddfe1b252195b5e27 \
    --hash=sha256:993837bbf0849e3b176e1ef6a50e9b8c2225e895501b85d56f4bb65a67f5ea25 \
    --hash=sha256:9a5f0cf8b95c0c058e413679a650f70dcc97764ccb2a6d5ccc6b08d44c9b334c \
    --hash=sha256:9f2839396d21d5537bc9ff53772d44db39b0efb6bf8b6cac709170483df53a5b \
    --hash=sha256:b8ba4b4ee3addc26bc595a51b6240b05a80e254b946d624fff6506439bc323d1 \
    --hash=sha256:bb6d90180eff72fc5a30099c442b8b0b5a620e84bf03ef32a55e3f7bd543f32e \
    --hash=sha256:c3d778304209cc39f8287da22f2180f34d2c2ee46cd55abd82e48178841b37b1 \
    --hash=sha256:c562bc316040097e21357e783286e5eca056a5b2750e89d9d75f9541c156b6dc \
    --hash=sha256:d114f9c0164df8fcd2880e4ba96986d7b0e7218f6984acc4989ff384c5d3d512 \
    --hash=sha256:d282b030ed5c736e4cdb1713a0c4fad7027f4e3959dc4b8fdb7c75042d83ed1b \
    --hash=sha256:d8c73fe0ec57a0e4fdf5d2728b5e18b63980f55f1baf51b6bac6a73e8cbb7186 \
    --hash=sha256:e5c8f4198e25bc4b0e4a884377e0c0e46ca273993679e3bcc212ef96d4211b83 \
    --hash=sha256:e7f1dcc0e8c3e18fa2fddca4aecdf71c5651555a8dc9a0cd3a1d164cbce6cb35 \
    --hash=sha256:ea3b61bff995de49b07331d1081e0056ea29901d3e995aa989073fe2b1be0cb7 \
    --hash=sha256:ea5f987b4da530822fa797cf2f010193be77ea9e232d07454e3194531edd8e58 \
    --hash=sha256:f91b16e73eca996f86d1943be3b2c2b679b03e068ed8c82a5506c1e65766e4a6
numpy==1.18.1 \
    --hash=sha256:1786a08236f2c92ae0e70423c45e1e62788ed33028f94ca99c4df03f5be6b3c6 \
    --hash=sha256:17aa7a81fe7599a10f2b7d95856dc5cf84a4eefa45bc96123cbbc3ebc568994e \
    --hash=sha256:20b26aaa5b3da029942cdcce719b363dbe58696ad182aff0e5dcb1687ec946dc \
    --hash=sha256:2d75908ab3ced4223ccba595b48e538afa5ecc37405923d1fea6906d7c3a50bc \
    --hash=sha256:39d2c685af15d3ce682c99ce5925cc66efc824652e10990d2462dfe9b8918c6a \
    --hash=sha256:56bc8ded6fcd9adea90f65377438f9fea8c05fcf7c5ba766bef258d0da1554aa \
    --hash=sha256:590355aeade1a2eaba17617c19edccb7db8d78760175256e3cf94590a1a964f3 \
    --hash=sha256:70a840a26f4e61defa7bdf811d7498a284ced303dfbc35acb7be12a39b2aa121 \
    --hash=sha256:77c3bfe65d8560487052ad55c6998a04b654c2fbc36d546aef2b2e511e760971 \
    --hash=sha256:9537eecf179f566fd1c160a2e912ca0b8e02d773af0a7a1120ad4f7507cd0d26 \
    --hash=sha256:9acdf933c1fd263c513a2df3dceecea6f3ff4419d80bf238510976bf9bcb26cd \
    --hash=sha256:ae0975f42ab1f28364dcda3dde3cf6c1ddab3e1d4b2909da0cb0191fa9ca0480 \
    --hash=sha256:b3af02ecc999c8003e538e60c89a2b37646b39b688d4e44d7373e11c2debabec \
    --hash=sha256:b6ff59cee96b454516e47e7721098e6ceebef435e3e21ac2d6c3b8b02628eb77 \
    --hash=sha256:b765ed3930b92812aa698a455847141869ef755a87e099fddd4ccf9d81fffb57 \
    --hash=sha256:c98c5ffd7d41611407a1103ae11c8b634ad6a43606eca3e2a5a269e5d6e8eb07 \
    --hash=sha256:cf7eb6b1025d3e169989416b1adcd676624c2dbed9e3bcb7137f51bfc8cc2572 \
    --hash=sha256:d92350c22b150c1cae7ebb0ee8b5670cc84848f6359cf6b5d8f86617098a9b73 \
    --hash=sha256:e422c3152921cece8b6a2fb6b0b4d73b6579bd20ae075e7d15143e711f3ca2ca \
    --hash=sha256:e840f552a509e3380b0f0ec977e8124d0dc34dc0e68289ca28f4d7c1d0d79474 \
    --hash=sha256:f3d0a94ad151870978fb93538e95411c83899c9dc63e6fb65542f769568ecfa5 \
    # via scipy
scipy==1.4.1 \
    --hash=sha256:00af72998a46c25bdb5824d2b729e7dabec0c765f9deb0b504f928591f5ff9d4 \
    --hash=sha256:0902a620a381f101e184a958459b36d3ee50f5effd186db76e131cbefcbb96f7 \
    --hash=sha256:1e3190466d669d658233e8a583b854f6386dd62d655539b77b3fa25bfb2abb70 \
    --hash=sha256:2cce3f9847a1a51019e8c5b47620da93950e58ebc611f13e0d11f4980ca5fecb \
    --hash=sha256:3092857f36b690a321a662fe5496cb816a7f4eecd875e1d36793d92d3f884073 \
    --hash=sha256:386086e2972ed2db17cebf88610aab7d7f6e2c0ca30042dc9a89cf18dcc363fa \
    --hash=sha256:71eb180f22c49066f25d6df16f8709f215723317cc951d99e54dc88020ea57be \
    --hash=sha256:770254a280d741dd3436919d47e35712fb081a6ff8bafc0f319382b954b77802 \
    --hash=sha256:787cc50cab3020a865640aba3485e9fbd161d4d3b0d03a967df1a2881320512d \
    --hash=sha256:8a07760d5c7f3a92e440ad3aedcc98891e915ce857664282ae3c0220f3301eb6 \
    --hash=sha256:8d3bc3993b8e4be7eade6dcc6fd59a412d96d3a33fa42b0fa45dc9e24495ede9 \
    --hash=sha256:9508a7c628a165c2c835f2497837bf6ac80eb25291055f56c129df3c943cbaf8 \
    --hash=sha256:a144811318853a23d32a07bc7fd5561ff0cac5da643d96ed94a4ffe967d89672 \
    --hash=sha256:a1aae70d52d0b074d8121333bc807a485f9f1e6a69742010b33780df2e60cfe0 \
    --hash=sha256:a2d6df9eb074af7f08866598e4ef068a2b310d98f87dc23bd1b90ec7bdcec802 \
    --hash=sha256:bb517872058a1f087c4528e7429b4a44533a902644987e7b2fe35ecc223bc408 \
    --hash=sha256:c5cac0c0387272ee0e789e94a570ac51deb01c796b37fb2aad1fb13f85e2f97d \
    --hash=sha256:cc971a82ea1170e677443108703a2ec9ff0f70752258d0e9f5433d00dda01f59 \
    --hash=sha256:dba8306f6da99e37ea08c08fef6e274b5bf8567bb094d1dbe86a20e532aca088 \
    --hash=sha256:dc60bb302f48acf6da8ca4444cfa17d52c63c5415302a9ee77b3b21618090521 \
    --hash=sha256:dee1bbf3a6c8f73b6b218cb28eed8dd13347ea2f87d572ce19b289d6fd3fbc59
six==1.14.0 \
    --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \
    --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c \
    # via anytree

[wwuck]

I have submitted a PR fixing this issue at #1065

[atugushev]

Hello @wwuck,

Thanks for the issue!

I hit this issue when trying to fix Arkq/flake8-requirements#14. It would appear that pip itself does not follow this part of the spec correctly, as it seems to be fine processing the existing (incorrect) generated output from pip-compile.

That's because pip joins lines at first and then ignores comments. See the code.

Just wondering why would anyone strip comments from requirements.txt?

[atugushev]

I believe this is a problem because the requirements file format spec says:

Comments are stripped before line continuations are processed.

That's odd. Probably this is a bug in the documentation since it's done the opposite in the code.

I've checked how it works in python and bash, comments there stripped after line continuations are processed, otherwise, the parser wouldn't allow you to add a comment after line continuation. See:

Python

ok = \
    "OK" \
    # OK

Bash

echo \
    OK \
    # OK

[atugushev]

It's definitely a bug in the pip doc. See the original PR where "Comments are stripped before line continuations are processed" appeared and note the order of preprocessors in preprocess function pypa/pip#3125.

Would you like to create an issue/PR on pip's issue tracker?

[wwuck]

Thanks for the analysis. I've created an issue at pypa/pip#7728.

[AndydeCleyre]

Motion to close this as invalid?

[atugushev]

I'd keep it open until pip maintainers respond to pypa/pip#7728.

[atugushev]

Close based on pypa/pip#7780 approval. Thanks for the issue nonetheless!

[pradyunsg]

@atugushev You didn't close this. :)

[atugushev]

@pradyunsg thank you for the reminder :)