Release 2.2 #40
Merged
Release 2.2 #40
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
📝🎨 Add Bioconda project url to setup.py 🎨 Add toast when browser storage limits are reached (issue #11) :art::memo: Move favicon to automappa/assets :arrow_up::art: Add geom-median to environment.yml :art: Add get_clusters_geom_medians(...) to celery tasks :arrow_up::art: Bump VERSION to 2.2.0 :see_no_evil: Add data to .gitignore
…bitmq) for task-queue
Add TODOs for WIP next steps... To be checked before merge into develop
🎨🐛 Prevent samples-store data from being overwritten on initial callback 🎨 samples-store data now persistent with local session 🎨 Refactor some variables in MAG refinement 🎨 WIP
… MetaData.table.keys() Should now be able to 🔥 remove all of the data stores to track the fileupload table_id... 🔥
🐳🌻🐘🎨 Add docker-services: celery, flower, prometheus, grafana, redis, rabbitmq 🎨🔥 Remove environment/*.env files. Consolidate to .env with corresponding prefixes 🎨 Change pydantic models built on BaseSettings in automappa.settings to use only .env with env_prefix 🎨🌻 Add working configuration of celery task-queue with redis backend and rabbitMQ broker 📝 Add resources/references for misc. links used during debugging/troubleshooting ⬆️ Add flower. Move celery[redis] to pip install section of environment.yml 🎨 black formatting to serializers and change server.upload_folder root to server.root_upload_folder 🎨 blak formatting to home.py
🎨 dbc.Select(...) are now populated with values from db tables 🐛🎨 Investigate whether serializer.get_uploaded_files_table() is functional (TODO) 🐳 Add commented out arg to create_engine(..., pre_pool_ping: Optional[bool]) and db settings 🐳 Add PRE_POOL_PING to .env
🔥 Remove (now deprecated) stores 🎨 Add serialization of a hash-refinement table when hash-binning file is uploaded (respective to hash-binning) 🎨 Remove --debug arg from parser. Move statement to .env (now handle read with automappa.settings.server.debug 🔥 remove unused imports in index.py 🎨 Move samples stores from home.py to index.py Add dbc.InputGroup(...) with selects for binning, markers and metagenome 🎨 Add dbc.Button(...) to save table-ids in 'selected-tables-store' to be read from mag_refinement.py 🎨🐛 Continue refactor of get_scatterplot_2d(...) to use other embedding methods 🎨 Add get_marker_symbols(...) to tasks.py and automappa.utils.markers (WIP currently being used to *expensively* generate symbols during 2d scatterplot creation
🔥 Remove unused (commented out) argparse args 🔥🎨🐳 Update automappa-web service command to not use removed argparse args
Will still need to ensure this is working once celery tasks are implemented
🐛🔥 Fix sankey diagram subsetting bug where selections would not render 🎨 Change MAG-specific completeness/purity boxplot to barplot 🎨 Rename save_to_db(...) to file_to_db(...) in serializer 🎨 Add metric_barplot(...) func to automappa.utils.figures
- 🐳🎨 Add mem_limit: 4GB to automappa-web service - 🎨🐛 Remove ability for user to navigate to other tabs when no data has been uploaded/selected - 🎨 Disable 'Refine mags' button if data has not been selected - 🎨 Add DataTable(...) to visualize the currently loaded datasets - 🎨 Change server settings import so it is clear that the server setting are imported and not the server - 🎨🥕 Silence numba and h5py loggers in mag_refinement.py (beginnings of celery task-queue tasks implementation) - 🎨 Add logger to automappa.tasks - 🐛 Fix logger emitted message during refinement data serialization to postgres db
…g methods 🎨🐛 Need to replace kmers.embed(...) method in tasks.py to include n_jobs=1 when autometa 2.0.4 release is out 🎨🥦 Add celeryconfig 🎨🥦 Add working kmer freq. analysis parallelize pipeline for use in task-queue 🎨 Add use of pydantic.BaseModel in models.py of SampleTables, KmerTable, AnnotationTable and misc. others (WIP) 🎨 Add construction of SeqIO.SeqRecords when retrieving metagenome from table using new func, get_metagenome_seqrecords(...) 🎨 Refactor get_contig_marker_counts(...) 🎨 Refactor get_marker_symbols(...) in markers.py 🎨🔥 Remove hard-coded values in automappa.utils.figures.get_embedding_traces_df(...) 🎨 Parse selected_data_tables as SampleTables.parse_raw(selected_data_tables) for shared data 🎨 Add tasks button to home.py 🎨 Replace typehints for selected_data_tables to Json[SampleTables]
🎨 Fetch data from postgres service 🎨🥦 Retrieve scatterplot-2d coordinates from celery task-queue results (selected from auto-generated dropdown) 🎨🌻 Add flower configuration env variable to settings.py 🎨 generate product of all combinations of kmer pipeline params (kmer_size, norm_method, embed_method) for 2d-coords retrieval 🎨🐎🐛 Retrieve mag summary data from SampleTables properties instead of using get_table(...) 🔥🐛 Remove default value of 'cluster' for mag_summary_cluster_col_dropdown 🎨🔥 Remove unnecessary nested lists of Input(...) in mag_summary callbacks 🎨 Add long_callback_manager to app (WIP: currently not being used) 🎨 Add handling of kmer param titles and metagenome annotation titles to format_axis_title(...) in figures.py 🎨🐳🐰 Change rabbitmq image from latest to 3.10.1-management-alpine (w/port) to use amqp management UI
📝 Move misc. markdown docs to docs directory
🐳 Ignore .vscode for docker context 🐳 Install mamba and prune prior to env update
💚🐎 Add dash-extensions to enable caching 🔥 Reduce kmer tasks to am_clr norm method and kmer_size of 5 (bhsne, densmap and umap embeddings)
Turn off debug mode by default
🔥 Change clone of `home-tab` branch to `develop` branch
🐳📺🎨 Automappa web app with home tab for data upload - 🥦 Add task-queue for pre/post-processing tasks with celery - 🟥 Add Redis service for celery backend - 🐰 Add RabbitMQ service for celery broker - 🐘 Add PostgreSQL for data ingestion - 🌷 Add Flower for celery task-queue monitoring - 🐍🎨 Add Pydantic for data models - 🐳 Add docker-compose file to orchestrate Automappa services
* 🎨 Apply black formatting
* 🎨⬆️🤠 refactor to use dash-extensions DashBlueprints
* Significantly simplified code base with modular components that are each passed their own page's app (DashBlueprint)
* components and layouts are standardized to use render function that may receive the app with or without arguments
* 🔥 Remove (now) unnecessary index.py
* ⬆️ Pin dash-extensions to first major release
* ⬆️ Pin dash to 2.10
* ⬆️ Add dash-mantine-components
* ⬆️ Add dash-iconify
* 📝 Add contributing docs
* 🎨🐛📝🐳 Fix docs, prop id and docker specs
* 🐳 Add quotes around ports per docker-compose spec
* 📝🎨🔥 add Development resources
🎨🔥 Add icons to not found page and remove unnecessary text
* 🎨 Refactor mag refinement page components
* 🎨 Refactor marker symbols legend to use icons rather than naming shapes
* 🎨 Restyle save selections and settings buttons
* 🎨 Refactor mag-refinement offsettings canvas components
* 🎨 Add minimum example contig_cytoscape component
* 🎨 Add specific backend configurations
* 🎨 Add REDIS_BACKEND details to .env
* 🎨 Add specific backend params to Serverside(..., backend=...) from automappa.data.db import {redis,file_system}_backend
* 📝🔥 Refactor dash components to DashProxy components
* ⬆️ pin dash-mantine-components to 0.12 in env using pip rather than conda
* 🎨 Add icons in page registry with blueprints
* 🎨 Add coverage range slider component
* 🎨 Add Refinement model to track contig cluster refinements
* 🎨 Implement data source to connect MAG refinements table
* 🎨 Add methods to get latest user refinements contig headers
* 🎨 Add interaction b/w coverage range slider and scatterplot2d
* 🎨 Connect download refinements button to data source
* 🎨 Hide MAG refinements switch is now disabled when there are no user refinements that have been made
* 🎨 Refinements table has been styled with new timestamp string and removed 'initial_cluster' column
* 🎨 Refinements table now only retrieves user refinements (i.e. excludes initial clusters)
* 🎨 Add refinements clear button
* Add working data source for completeness/purity boxplot
* 🎨 Add minimal working AgGrid for MAG Summary
* 🎨 Replace dbc.Offcanvas with dmc.Drawer
* 🎨 Replace many of the Dropdowns in Offcanvas with RadioGroup Radio options (less clicks for the user)
* 🥕🐰 Add logic for modular task-queue
* 📝 Add documentation for tasks in CONTRIBUTING and reference tasks README.md
* 🥕 Add minimum working example for task submission, progress display for a task status badge
* 🥕 Add task discovery for home page task in celeryconfig.py
* 🐳🐍 Replace base image of miniforge with mambaforge
* 🥕🎨 Notification system tells user of current state of tasks processing
* 🐛🎨 Move dcc.Store(...) within respective du.Upload component
* 🎨 Add tasks store for getting tasks statuses
* 🐰🥕 Add tasks in home for handling sample data ingestion
* 🎨 Add mag length sum computations
* 🎨 Add MIMAG cluster count computations
* 🎨 Add metadata to sample data card info
* 🐛 Replace na values from cluster col during contig loader pre-processing
* 🎨🔥 Add sample removal feature
* 🎨 Affix settings and save mag buttons
evanroyrees
added
documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release 2.2
A complete refactor for improved maintainability, code comprehension and most importantly performance.
New features for quickly exploring your environmental samples across multiple datasets. Automappa will persist your changes, so no longer must you spend hours painstakingly curating your genomes only for them to be lost by a pesky browser crash!
A shortlist of nice features with this update 🎈