Skip to content

Releases: qiita-spots/qiita

2024.07

16 Jul 12:03
Compare
Choose a tag to compare

Deployed on July 16th, 2024

  • On June 14th, 2024 we modified the SPP to use "fastp & minimap2 against GRCh38.p14 + Phi X 174 + T2T-CHM13v2.0, then Movi against GRCh38.p14, T2T-CHM13v2.0 + Human Pangenome Reference Consortium release 2023" to filter human-reads.
  • Full refactor of the DB patching system to make sure that a new production deployment has a fully empty database.
  • Fully removed Qiimp from Qiita.
  • Users can now add ORCID, ResearchGate and/or GoogleScholar information to their profile and the creation (registration) timestamp is kept in the database. Thank you @jlab.
  • Admins can now track and purge non-confirmed users from the database via the GUI (/admin/purge_users/). Thank you @jlab.
  • Added qiita.slurm_resource_allocations to store general job resource usage, which can be populated by qiita_db.util.update_resource_allocation_table.
  • Added qiita_db.util.resource_allocation_plot to generate different models to allocate resources from a given software command based on previous jobs, thank you @Gossty !
  • The stats page map can be centered via the configuration file; additionally, the Help and Admin emails are defined also via the configuration files, thank you @jlab !
  • Sequel IIe, Revio, and Onso are now valid instruments for the PacBio_SMRT platform.
  • Added current_human_filtering to the prep-information and human_reads_filter_method to the artifact to keep track of the method that it was used to human reads filter the raw artifact and know if it's up to date with what is expected via the best practices.
  • Added reprocess_job_id to the prep-information so we keep track if a preparation has been reprocessed with another job.
  • Other general fixes, like #3385, #3397, #3399, #3400, #3409, #3410.

2024.02

27 Feb 12:37
Compare
Choose a tag to compare

Deployed on February 27th, 2024

  • Default workflows now accept commands with multiple inputs.
  • The loading time of the main study page was improved #3350.
  • SPP improvements - mainly @charles-cowart, thank you! Errors are now show to the user in the GUI #127, admins can restart jobs #129, adapter-trimmer files now are stored and their sequence counts are part of the prep-info #126, and support for per instrument/data-type configuration #123.
  • The internal Sequence Processing Pipeline is now using the https://www.gencodegenes.org human transcripts v44 for Metatranscriptomic data - additional to the human pan-genome reference, with the GRCh38 genome + PhiX and T2T-CHM13v2.0 genome - for human host filtering.
  • Added a command to qp-woltka: 'Calculate RNA Copy Counts'.
  • Other fixes - mainly by @sjanssen2, thank you!: #3345,#3224, #3357, #3358, #3359, #3362, #3364.

2023.12

08 Jan 13:21
Compare
Choose a tag to compare

Version 2023.12

Deployed on January 8th, 2024

  • The sample and preparation information pages will display the timestamp of their last update.
  • Added a ProcessingJob.complete_processing_job method to retrieve the job that is completing the current job.
  • Added a ProcessingJob.trace method to trace all the jobs of a processing_job.
  • Analyses now accept SLURM reservation's via the GUI; this will be helpful for workshops or classes.
  • Admins can now add per-user-level SLURM submission parameters via the DB; this is helpful to prioritize wet-lab and admin jobs.
  • Workflow definitions can now use sample or preparation information columns/values to differentiate between them.
  • Updated the Adapter and host filtering plugin (qp-fastp-minimap2) to v2023.12 addressing a bug in adapter filtering; more information.
  • Other fixes: 3334, 3338. Thank you @sjanssen2.
  • The internal Sequence Processing Pipeline is now using the human pan-genome reference, together with the GRCh38 genome + PhiX and CHM13 genome for human host filtering.

2023.10

16 Oct 12:33
ac62aba
Compare
Choose a tag to compare
  • Added a new notebooks folder to the repository to allocate the resource-allocation code and plotting.
  • Allowed dynamic resource allocations for time (before it was only for memory), and allowed complex formulas (via eval).
  • Deployed the new resource allocation formulas. This included changes on how the ProcesingJob.shape method calculates its variables, which include: (a) build_analysis_files: only use the biom file (vs. all of them); and (b) the SPP counts the input-file number of lines know the number of samples.
  • qp-woltka generates a new file: coverates.tgz, which contains: (a) artifact.cov: the coverages of this artifact, can be used with other coverages to generate combined coverages; (b) coverage_percentage.txt: the total percentage coverage per-genome of this artifact; and (c) coverages folder: the per-sample coverage.
  • The SPP now supports replicates within the same lane/run and creates multiple preparations as needed.
  • The SPP automatically adds and executes the default pipeline after it is done generating and linking the raw data.
  • Other fixes: #3300, metapool #135, metapool #136, qtp-sequencing #47.

2023.06

20 Jun 11:46
4492c3d
Compare
Choose a tag to compare
  • EBI-ENA submissions require a sample metadata column: geographic location (country and/or sea), Qiita will automatically rename country to satisfy this requirement.
  • The qp-knight-lab-processing was refactored for better performance and testing; mainly contributed by @charles-cowart.
  • Admin Sample Validation now uses tube_id; mainly contributed by @CatFish47.
  • Registered users cannot start sample deletion jobs within public studies.
  • The qp-knight-lab-processing now links the SPP job to the preparation it creates and links the created files to that preparation. Note that no files are being moved to the upload folder.
  • Fixed the following issues #3227, #3279, #3227.
  • Added ProcessingJob.external_id to status-report messages generated, and added sending an email to our system-admin when a (wet-lab) admin job errors.

2023.05

14 May 12:16
9478d32
Compare
Choose a tag to compare

NOTE: Human-associated shotgun samples have been determined to leak human DNA, even with state of the art human read filtering mechanisms in place. To address this critical privacy concern as soon as possible, we are limiting per-sample FASTQ downloads to admins and study owners for human-associated shotgun samples, and in parallel we are (a) working with EBI-ENA to address submitted data and (b) updating Qiita to only allow download of sequence data that positively matches to a microbial specific database.

  • Added "Artifact.has_human" to check if an artifact has samples that can potentially contain human sequences.

2023.03

26 Mar 23:33
b8e151e
Compare
Choose a tag to compare
  • Added "Sample Validation" for all Admin levels.
  • Tools and Plugins can add preparation information via the endpoint: /qiita_db/prep_template/.
  • Users can now access specific preparations via the URL by adding it as a parameter: /study/description/<study_id>?prep_id=<prep_id>.
  • The internal Sequence Processing Pipeline now automatically inserts new BLANKs to the sample information file, adds the preparation information file to Qiita, and keeps a record of which preparations were inserted to which studies.
  • Added the possibility of processing BIOM artifacts in the processing pipeline.
  • Added qiime2.2023.2 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
  • Added q2-greengeenes2 to the system, specifically actions: filter_features and non_v4_16s. This is the first processing (vs. analysis) plugin added to the system.
  • Fixed the diversity pcoa_biplot functionality and added the emperor biplot action.
  • The system can now validate QIIME 2 FeatureData[Sequences] via qtp-diversity.
  • Fixed the following issues: #3249, #3264, #3245, #3243.

2023.02

24 Feb 15:41
Compare
Choose a tag to compare
  • Added "Software and Data Licensing" to the Qiita FAQ page.
  • External resources can now add new sample metadata categories to a study via
    the /api/v1/study/ endpoint.
  • Added preparation-id to the GUI list of artifacts used in an analysis.
  • Added automatic lower-casing to INSDC null values #3246.
  • ArtifactHandler now returns the file size and full path of the files available in
    an Artifact. This change had two consequences: (1) the plugins now can control
    their behavior based on the file sizes, and (2) all plugins had to be updated to use this new feature.
  • Added qiita_client.artifact_and_preparation_files
    to help plugins filter per_sample_FASTQ based on size and ignore small file sizes.
  • Added qiime2.2022.11 to the system; which updated these plugins: qp-qiime2, qtp-biom, qtp-diversity, qtp-visualization.
  • Added q2-mislabeled and composition's ancom to the qiime2.2022.11 deployment.
  • Added support for Amplicon data to the internal Sequence Processing Pipeline.

2022.11

18 Nov 18:45
684a7a4
Compare
Choose a tag to compare
  • Adding new Woltka v0.1.4 and WoLr2 - WoLr1 and RS210 (updated version of rep200) are available but not default.
  • Update qp-woltka so it runs faster, contribution with @wasade. Add newest improvements for woltka processing so it runs faster.
  • Fix qp-qiime2, "filter features" bug that expects a "feature metadata" value from the dropdown.
  • Avoid having multiple git version commands running in the system - this in theory should fix some of the slow downs.
  • Fix sample_name in per preparation sample information files.
  • Now the sequencing artifacts (like per-sample-FASTQ, FASTA, etc) accept log files.
  • SortMeRNA via qp-meta will generate processing logs and will be added to their artifacts - note that this will only be available in newer runs AKA you can delete the outputs of past runs and rerun it.
  • Added new "Admin Sample Validation" to validate a list of sample names against what's available in a study, contributed by @sarayupai.

2022.09

15 Sep 18:34
Compare
Choose a tag to compare
  • Moved Qiita's code base and plugins to SLURM (from Torque). The plugins updated are: qiita-spots/qp-woltka, biocore/mg-scripts, qiita-spots/qp-spades, qiita-spots/qp-meta, qiita-spots/qp-fastp-minimap2.
  • Pinned the paramiko version to < 2.9 as newer versions were causing issues with older systems.
  • Pinned the scipy version to < 1.8 to avoid issues with the biom-format library.
  • Updates to the INSTALL instructions (thank you @aliu104 !)