Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐳📺🎨 Automappa web app with server upload and background pre-/post-processing services #35

Merged
merged 38 commits into from
Aug 3, 2022

Commits on Dec 19, 2021

  1. Configuration menu
    Copy the full SHA
    406d1a7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5e05bf9 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2021

  1. Configuration menu
    Copy the full SHA
    49ff515 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ffcfc48 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2e3f695 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    8e0b80c View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2022

  1. Configuration menu
    Copy the full SHA
    933ceeb View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2022

  1. Configuration menu
    Copy the full SHA
    c096c3e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2bc5c29 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a02c212 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2022

  1. 🎨📝✅💚⬆️ Add geom_median to env, move favicon assets

    📝🎨 Add Bioconda project url to setup.py
    🎨 Add toast when browser storage limits are reached (issue #11)
    :art::memo: Move favicon to automappa/assets
    :arrow_up::art: Add geom-median to environment.yml
    :art: Add get_clusters_geom_medians(...) to celery tasks
    :arrow_up::art: Bump VERSION to 2.2.0
    :see_no_evil: Add data to .gitignore
    evanroyrees committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    ed8c0d1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f79dee0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    399c2f0 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2022

  1. 🎨🐛 WIP beginning of using postgres db

    Add TODOs for WIP next steps... To be checked before merge into develop
    evanroyrees committed Apr 14, 2022
    Configuration menu
    Copy the full SHA
    1df246b View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2022

  1. Refactor callback to viz DataTable and store samples-store State

    🎨🐛 Prevent samples-store data from being overwritten on initial callback
    🎨 samples-store data now persistent with local session
    🎨 Refactor some variables in MAG refinement
    🎨 WIP
    evanroyrees committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    8a54e12 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7b5f2d3 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2022

  1. Configuration menu
    Copy the full SHA
    991a45f View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2022

  1. Configuration menu
    Copy the full SHA
    3013b62 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2022

  1. 🎨 WIP now fetching all fileuploads and creating files datatable using…

    … MetaData.table.keys()
    
    Should now be able to 🔥 remove all of the data stores to track the fileupload table_id... 🔥
    evanroyrees committed Apr 22, 2022
    Configuration menu
    Copy the full SHA
    ea83e85 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d0f943f View commit details
    Browse the repository at this point in the history
  3. 🔥🐳🎨🌻🐘 Add docker-services, rm environment/

    🐳🌻🐘🎨 Add docker-services: celery, flower, prometheus, grafana, redis, rabbitmq
    🎨🔥 Remove environment/*.env files. Consolidate to .env with corresponding prefixes
    🎨 Change pydantic models built on BaseSettings in automappa.settings to use only .env with env_prefix
    🎨🌻 Add working configuration of celery task-queue with redis backend and rabbitMQ broker
    📝 Add resources/references for misc. links used during debugging/troubleshooting
    ⬆️ Add flower. Move celery[redis] to pip install section of environment.yml
    🎨 black formatting to serializers and change server.upload_folder root to server.root_upload_folder
    🎨 blak formatting to home.py
    evanroyrees committed Apr 22, 2022
    Configuration menu
    Copy the full SHA
    cd6d4f8 View commit details
    Browse the repository at this point in the history
  4. Add dbc.Select to home for choosing metagenome annotations (WIP)

    🎨 dbc.Select(...) are now populated with values from db tables
    🐛🎨 Investigate whether serializer.get_uploaded_files_table() is functional (TODO)
    🐳 Add commented out arg to create_engine(..., pre_pool_ping: Optional[bool]) and db settings
    🐳 Add PRE_POOL_PING to .env
    evanroyrees committed Apr 22, 2022
    Configuration menu
    Copy the full SHA
    6e92c24 View commit details
    Browse the repository at this point in the history
  5. 🎨 WIP refinement page reading user uploaded/selected data

    🔥 Remove (now deprecated) stores
    🎨 Add serialization of a hash-refinement table when hash-binning file is uploaded (respective to hash-binning)
    🎨 Remove --debug arg from parser. Move statement to .env (now handle read with automappa.settings.server.debug
    🔥 remove unused imports in index.py
    🎨 Move samples stores from home.py to index.py
    Add dbc.InputGroup(...) with selects for binning, markers and metagenome
    🎨 Add dbc.Button(...) to save table-ids in 'selected-tables-store' to be read from mag_refinement.py
    🎨🐛 Continue refactor of get_scatterplot_2d(...) to use other embedding methods
    🎨 Add get_marker_symbols(...) to tasks.py and automappa.utils.markers (WIP currently being used to *expensively* generate symbols during 2d scatterplot creation
    evanroyrees committed Apr 22, 2022
    Configuration menu
    Copy the full SHA
    c6fd9da View commit details
    Browse the repository at this point in the history
  6. 🎨 Move port and host argparse args to .env

    🔥 Remove unused (commented out) argparse args
    🔥🎨🐳 Update automappa-web service command to not use removed argparse args
    evanroyrees committed Apr 22, 2022
    Configuration menu
    Copy the full SHA
    dc5665d View commit details
    Browse the repository at this point in the history
  7. 📝 Update TODO list

    evanroyrees committed Apr 22, 2022
    Configuration menu
    Copy the full SHA
    e54108c View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2022

  1. 🐳📺 provisioned Grafana within docker-compose.

    Will still need to ensure this is working once celery tasks are implemented
    evanroyrees committed Apr 23, 2022
    Configuration menu
    Copy the full SHA
    d35a6cc View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2022

  1. 🎨🐛 Connect mag_summary.layout callbacks to postgres db

    🐛🔥 Fix sankey diagram subsetting bug where selections would not render
    🎨 Change MAG-specific completeness/purity boxplot to barplot
    🎨 Rename save_to_db(...) to file_to_db(...) in serializer
    🎨 Add metric_barplot(...) func to automappa.utils.figures
    evanroyrees committed Apr 25, 2022
    Configuration menu
    Copy the full SHA
    ec9989b View commit details
    Browse the repository at this point in the history
  2. 🎨 Beginning of celery task-queue implementation

    - 🐳🎨 Add mem_limit: 4GB to automappa-web service
    - 🎨🐛 Remove ability for user to navigate to other tabs when no data has been uploaded/selected
    - 🎨 Disable 'Refine mags' button if data has not been selected
    - 🎨 Add DataTable(...) to visualize the currently loaded datasets
    - 🎨 Change server settings import so it is clear that the server setting are imported and not the server
    - 🎨🥕 Silence numba and h5py loggers in mag_refinement.py (beginnings of celery task-queue tasks implementation)
     - 🎨 Add logger to automappa.tasks
    - 🐛 Fix logger emitted message during refinement data serialization to postgres db
    evanroyrees committed Apr 25, 2022
    Configuration menu
    Copy the full SHA
    bbc93e9 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2022

  1. Configuration menu
    Copy the full SHA
    9917443 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2022

  1. 🎨 WIP Working task-queue with scatterplot-2d views including embeddin…

    …g methods
    
    🎨🐛 Need to replace kmers.embed(...) method in tasks.py to include n_jobs=1 when autometa 2.0.4 release is out
    🎨🥦 Add celeryconfig
    🎨🥦 Add working kmer freq. analysis parallelize pipeline for use in task-queue
    🎨 Add use of pydantic.BaseModel in models.py of SampleTables, KmerTable, AnnotationTable and misc. others (WIP)
    🎨 Add construction of SeqIO.SeqRecords when retrieving metagenome from table using new func, get_metagenome_seqrecords(...)
    🎨 Refactor get_contig_marker_counts(...)
    🎨 Refactor get_marker_symbols(...) in markers.py
    🎨🔥 Remove hard-coded values in automappa.utils.figures.get_embedding_traces_df(...)
    🎨 Parse selected_data_tables as SampleTables.parse_raw(selected_data_tables) for shared data
    🎨 Add tasks button to home.py
    🎨 Replace typehints for selected_data_tables to Json[SampleTables]
    evanroyrees committed May 5, 2022
    Configuration menu
    Copy the full SHA
    731ffc1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    977dbbf View commit details
    Browse the repository at this point in the history

Commits on May 16, 2022

  1. 🎨🙈 WIP for fetching/visualizing data using services

    🎨 Fetch data from postgres service
    🎨🥦 Retrieve scatterplot-2d coordinates from celery task-queue results (selected from auto-generated dropdown)
    🎨🌻 Add flower configuration env variable to settings.py
    🎨 generate product of all combinations of kmer pipeline params (kmer_size, norm_method, embed_method) for 2d-coords retrieval
    🎨🐎🐛 Retrieve mag summary data from SampleTables properties instead of using get_table(...)
    🔥🐛 Remove default value of 'cluster' for mag_summary_cluster_col_dropdown
    🎨🔥 Remove unnecessary nested lists of Input(...) in mag_summary callbacks
    🎨 Add long_callback_manager to app (WIP: currently not being used)
    🎨 Add handling of kmer param titles and metagenome annotation titles to format_axis_title(...) in figures.py
    🎨🐳🐰 Change rabbitmq image from latest to 3.10.1-management-alpine (w/port) to use amqp management UI
    evanroyrees committed May 16, 2022
    Configuration menu
    Copy the full SHA
    4810ba2 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2022

  1. 💚 Add test SampleTables model instance

    📝 Move misc. markdown docs to docs directory
    evanroyrees committed May 18, 2022
    Configuration menu
    Copy the full SHA
    ef3f205 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2022

  1. Configuration menu
    Copy the full SHA
    2a83bfa View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2022

  1. 🐳💚🐛 pin scipy to 1.8

    🐳 Ignore .vscode for docker context
    🐳 Install mamba and prune prior to env update
    evanroyrees committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    90494cd View commit details
    Browse the repository at this point in the history
  2. 🎨🐳🐎 Add dash-extensions

    💚🐎 Add dash-extensions to enable caching
    🔥 Reduce kmer tasks to am_clr norm method and kmer_size of 5 (bhsne, densmap and umap embeddings)
    evanroyrees committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    90b2990 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2022

  1. Update .env

    Turn off debug mode by default
    evanroyrees committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    df22b6e View commit details
    Browse the repository at this point in the history
  2. Update README.md

    🔥 Change clone of `home-tab` branch to `develop` branch
    evanroyrees committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    ebaccf4 View commit details
    Browse the repository at this point in the history