Fixing bug with slices landing on line ends (SCP-4648)

[bistline]

BACKGROUND & CHANGES

This fixes a bug where reading matrix slices that land exactly on a line ending result in an endless while loop that causes processing to hang. This also addresses a related issue where empty gene files named .json.gz were being created. Lastly, this removes the .gz suffix to rendered files to maintain parity with how SCP stores gzipped data in GCS buckets with Content-Encoding: gzip headers.

MANUAL TESTING

1. Pull branch, and either initialize scp-ingest-pipeline environment via source env/bin/active or by building the Docker container with docker build -t gcr.io/broad-singlecellportal-staging/ingest-pipeline:test-candidate .
2. Run export BYPASS_MONGO_WRITES=yes in the terminal (or Docker container) once the environment is set up
3. Run the following command:

python ingest_pipeline.py --study-id 5d276a50421aa9117c982845 --study-file-id 5dd5ae25421aa910a723a337 \
                             render_expression_arrays --matrix-file-path ../tests/data/expression_writer/slice_testing/expression_matrix.tsv \
                             --matrix-file-type dense \
                             --cluster-file ../tests/data/expression_writer/slice_testing/cluster.tsv \
                             --cluster-name 'Slice Example' --render-expression-arrays

4. Navigate to Slice_Example on the command line and validate that there are the following files: Adcy5.json, Agpat2.json, Agtr1.json, Aifm1.json, Apex1.json, Apoc3.json, Apoe.json and no empty file called '.json'

[codecov]

Codecov Report

Base: 66.75% // Head: 66.75% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (c87e651) compared to base (545c4a6).
Patch coverage: 66.66% of modified lines in pull request are covered.

Additional details and impacted files

@@               Coverage Diff               @@
##           development     #275      +/-   ##
===============================================
- Coverage        66.75%   66.75%   -0.01%     
===============================================
  Files               29       29              
  Lines             3974     3979       +5     
===============================================
+ Hits              2653     2656       +3     
- Misses            1321     1323       +2     

Impacted Files	Coverage Δ
ingest/ingest_files.py	85.50% <0.00%> (-0.80%)	⬇️
ingest/expression_writer.py	90.56% <70.00%> (+0.05%)	⬆️
ingest/writer_functions.py	97.50% <100.00%> (+0.03%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[eweitz]

Code looks good! Nice refinements.

[ehanna4]

Functional Review - Works as advertised 👍