diff --git a/README.md b/README.md index 2dfa785..5352992 100644 --- a/README.md +++ b/README.md @@ -83,9 +83,18 @@ quarto check > [!TIP] > If quarto is not installed, you can download the command-line interface from the [Quarto website][quarto-cli] for your operating system. +For PDF reports, you need to have a LaTeX distribution installed. This can be done with quarto using the following command: + +```bash +quarto install tinytex +``` + +> [!TIP] +> Also, you can add the `--quarto_checks` argument to the VueGen command to check and install the required dependencies automatically. + ### Docker -If you prefer not to install VueGen on your system, a pre-configured Docker container is available. It includes all dependencies, ensuring a fully reproducible execution environment. See the [Execution section](#execution) for details on running VueGen with Docker. The official Docker image is available at [quay.io/dtu_biosustain_dsp/vuegen][vuegen-docker-quay]. +If you prefer not to install VueGen on your system, a pre-configured Docker container is available. It includes all dependencies, ensuring a fully reproducible execution environment. See the [Execution section](#execution) for details on running VueGen with Docker. The official Docker images are available at [quay.io/dtu_biosustain_dsp/vuegen][vuegen-docker-quay]. The Dockerfiles to build the images are available [here][docker-folder]. ### Nextflow and nf-core @@ -94,7 +103,7 @@ VueGen is also available as a [nf-core][nfcore] module, customised for compatibi ## Execution > [!IMPORTANT] -> Here we use the `Earth_microbiome_vuegen_demo_notebook` directory and the `Earth_microbiome_vuegen_demo_notebook.yaml` configuration file as examples, which are available in the `docs/example_data` and `docs/example_config_files` folders, respectively. Make sure to clone this reposiotry to access these contents, or use your own directory and configuration file. +> Here we use the `Earth_microbiome_vuegen_demo_notebook` [directory][emp-dir] and the `Earth_microbiome_vuegen_demo_notebook.yaml` [configuration file][emp-config] as examples, which are available in the `docs/example_data` and `docs/example_config_files` folders, respectively. Make sure to clone this reposiotry to access these contents, or use your own directory and configuration file. Run VueGen using a directory with the following command: @@ -105,12 +114,41 @@ vuegen --directory docs/example_data/Earth_microbiome_vuegen_demo_notebook --rep > [!NOTE] > By default, the `streamlit_autorun` argument is set to False, but you can use it in case you want to automatically run the streamlit app. +### Folder structure + +Your input directory must follow a **nested folder structure**, where first-level folders are treated as **sections** and second-level folders as **subsections**, containing the components (plots, tables, networks, Markdown text, and HTML files). + +Here is an example layout: +``` +report_folder/ +├── section1/ +│ └── subsection1/ +│ ├── table.csv +│ ├── image1.png +│ └── chart.json +├── section2/ +│ ├── subsection1/ +│ │ ├── summary_table.xls +│ │ └── network_plot.graphml +│ └── subsection2/ +│ ├── report.html +│ └── summary.md +``` + +> [!WARNING] +> VueGen currently requires each section to contain at least one subsection folder. Defining only sections (with no subsections) or using deeper nesting levels (i.e., sub-subsections) will result in errors. In upcoming releases, we plan to support more flexible directory structures. + +The titles for sections, subsections, and components are extracted from the corresponding folder and file names, and afterward, users can add descriptions, captions, and other details to the configuration file. Component types are inferred from the file extensions and names. +The order of sections, subsections, and components can be defined using numerical suffixes in folder and file names. + It's also possible to provide a configuration file instead of a directory: ```bash vuegen --config docs/example_config_files/Earth_microbiome_vuegen_demo_notebook.yaml --report_type streamlit ``` +If a configuration file is given, users can specify titles and descriptions for sections and subsections, as well as component paths and required attributes, such as file format and delimiter for dataframes, plot types, and other details. + The current report types supported by VueGen are: - Streamlit @@ -130,7 +168,7 @@ Instead of installing VueGen locally, you can run it directly from a Docker cont docker run --rm \ -v "$(pwd)/docs/example_data/Earth_microbiome_vuegen_demo_notebook:/home/appuser/Earth_microbiome_vuegen_demo_notebook" \ -v "$(pwd)/output_docker:/home/appuser/streamlit_report" \ - quay.io/dtu_biosustain_dsp/vuegen:v0.3.1-docker --directory /home/appuser/Earth_microbiome_vuegen_demo_notebook --report_type streamlit + quay.io/dtu_biosustain_dsp/vuegen:v0.3.2-docker --directory /home/appuser/Earth_microbiome_vuegen_demo_notebook --report_type streamlit ``` ## GUI @@ -201,6 +239,9 @@ This introductory case study uses a predefined directory with plots, dataframes, 🔗 [![Open in Colab][colab_badge]][colab_link_intro_demo] +> [!NOTE] +> The [configuration file][predef-dir-config] is available in the `docs/example_config_files` folder, and the [directory][predef-dir] with example data is in the `docs/example_data` folder. + **2. Earth Microbiome Project Data** This advanced case study demonstrates the application of VueGen in a real-world scenario using data from the [Earth Microbiome Project (EMP)][emp]. The EMP is an initiative to characterize global microbial taxonomic and functional diversity. The notebook process the EMP data, create plots, dataframes, and other components, and organize outputs within a directory to produce reports. Report content and structure can be adapted by modifying the configuration file. Each report consists of sections on exploratory data analysis, metagenomics, and network analysis. @@ -209,6 +250,36 @@ This advanced case study demonstrates the application of VueGen in a real-world > [!NOTE] > The EMP case study is available online as [HTML][emp-html-demo] and [Streamlit][emp-st-demo] reports. +> The [configuration file][emp-config] is available in the `docs/example_config_files` folder, and the [directory][emp-dir] with example data is in the `docs/example_data` folder. + +**3. ChatBot Component** + +This case study highlights VueGen’s capability to embed a chatbot component into a report subsection, +enabling interactive conversations inside the report. + +Two API modes are supported: + +- **Ollama-style streaming chat completion** +If a `model` parameter is specified in the config file, VueGen assumes the chatbot is using Ollama’s [/api/chat endpoint][ollama_chat]. +Messages are handled as chat history, and the assistant responses are streamed in real time for a smooth and responsive experience. +This mode supports LLMs such as `llama3`, `deepsek`, or `mistral`. + +> [!TIP] +> See [Ollama’s website][ollama] for more details. + +- **Standard prompt-response API** +If no `model` is provided, VueGen uses a simpler prompt-response flow. +A single prompt is sent to an endpoint, and a structured JSON object is expected in return. +Currently, the response can include: + - `text`: the main textual reply + - `links`: a list of source URLs (optional) + - `HTML content`: an HTML snippet with a Pyvis network visualization (optional) + +This response structure is currently customized for an internal knowledge graph assistant, but VueGen is being actively developed +to support more flexible and general-purpose response formats in future releases. + +> [!NOTE] +> You can see a [configuration file example][config-chatbot] for the chatbot component in the `docs/example_config_files` folder. ## Web application deployment @@ -261,6 +332,7 @@ We appreciate your feedback! If you have any comments, suggestions, or run into [vuegen-pypi]: https://pypi.org/project/vuegen/ [vuegen-conda]: https://anaconda.org/bioconda/vuegen [vuegen-docker-quay]: https://quay.io/repository/dtu_biosustain_dsp/vuegen +[docker-folder]: https://github.com/Multiomics-Analytics-Group/nf-vuegen/tree/main/Docker [vuegen-license]: https://github.com/Multiomics-Analytics-Group/vuegen/blob/main/LICENSE [vuegen-class-diag-att]: https://raw.githubusercontent.com/Multiomics-Analytics-Group/vuegen/main/docs/images/vuegen_classdiagram_withattmeth.pdf [vuegen-docs]: https://vuegen.readthedocs.io/ @@ -268,6 +340,9 @@ We appreciate your feedback! If you have any comments, suggestions, or run into [ci-docs]: https://github.com/Multiomics-Analytics-Group/vuegen/actions/workflows/docs.yml [emp-html-demo]: https://multiomics-analytics-group.github.io/vuegen/ [emp-st-demo]: https://earth-microbiome-vuegen-demo.streamlit.app/ +[ollama_chat]: https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-chat-completion +[ollama]: https://ollama.com/ +[config-chatbot]: https://github.com/Multiomics-Analytics-Group/vuegen/blob/main/docs/example_config_files/Chatbot_example_config.yaml [issues]: https://github.com/Multiomics-Analytics-Group/vuegen/issues [pulls]: https://github.com/Multiomics-Analytics-Group/vuegen/pulls [vuegen-preprint]: https://doi.org/10.1101/2025.03.05.641152 @@ -279,8 +354,12 @@ We appreciate your feedback! If you have any comments, suggestions, or run into [nf-vuegen]: https://github.com/Multiomics-Analytics-Group/nf-vuegen/ [colab_badge]: https://colab.research.google.com/assets/colab-badge.svg [colab_link_intro_demo]: https://colab.research.google.com/github/Multiomics-Analytics-Group/vuegen/blob/main/docs/vuegen_basic_case_study.ipynb +[predef-dir-config]: https://github.com/Multiomics-Analytics-Group/vuegen/blob/main/docs/example_config_files/Basic_example_vuegen_demo_notebook_config.yaml +[predef-dir]: https://github.com/Multiomics-Analytics-Group/vuegen/blob/main/docs/example_data/Basic_example_vuegen_demo_notebook [colab_link_emp_demo]: https://colab.research.google.com/github/Multiomics-Analytics-Group/vuegen/blob/main/docs/vuegen_case_study_earth_microbiome.ipynb [emp]: https://earthmicrobiome.org/ +[emp-config]: https://github.com/Multiomics-Analytics-Group/vuegen/blob/main/docs/example_config_files/Earth_microbiome_vuegen_demo_notebook_config +[emp-dir]: https://github.com/Multiomics-Analytics-Group/vuegen/blob/main/docs/example_data/Earth_microbiome_vuegen_demo_notebook [st-cloud]: https://streamlit.io/cloud [stlite]: https://github.com/whitphx/stlite [st-forum-exe]: https://discuss.streamlit.io/t/streamlit-deployment-as-an-executable-file-exe-for-windows-macos-and-android/6812 diff --git a/docs/example_config_files/APIcall_example_config.yaml b/docs/example_config_files/APIcall_example_config.yaml index 96395ce..770cb1c 100644 --- a/docs/example_config_files/APIcall_example_config.yaml +++ b/docs/example_config_files/APIcall_example_config.yaml @@ -1,11 +1,44 @@ report: title: APICall example - description: An APICall exaple. + description: An APICall example. sections: - title: APICall test subsections: - title: JSONPlaceholder test components: - - title: JSONPlaceholder component + - title: GET request component_type: apicall api_url: https://jsonplaceholder.typicode.com/todos/1 + method: GET + - title: POST request + component_type: apicall + api_url: https://jsonplaceholder.typicode.com/todos + method: POST + request_body: | + { + "userId": 1, + "title": "Go running", + "completed": false + } + - title: PUT request + component_type: apicall + api_url: https://jsonplaceholder.typicode.com/todos/10 + method: PUT + request_body: | + { + "userId": 1, + "title": "Play the guitar", + "completed": true + } + - title: PATCH request + component_type: apicall + api_url: https://jsonplaceholder.typicode.com/todos/10 + method: PATCH + request_body: | + { + "title": "Go for a hike" + } + - title: DELETE request + component_type: apicall + api_url: https://jsonplaceholder.typicode.com/todos/10 + method: DELETE \ No newline at end of file diff --git a/src/vuegen/config_manager.py b/src/vuegen/config_manager.py index 9ebab17..dac12b4 100644 --- a/src/vuegen/config_manager.py +++ b/src/vuegen/config_manager.py @@ -561,13 +561,24 @@ def _create_apicall_component(self, component_data: dict) -> r.APICall: APICall An APICall object populated with the provided metadata. """ + request_body = component_data.get("request_body") + parsed_body = None + if request_body: + try: + parsed_body = json.loads(request_body) + except json.JSONDecodeError as e: + self.logger.error(f"Failed to parse request_body JSON: {e}") + raise ValueError(f"Invalid JSON in request_body: {e}") + return r.APICall( title=component_data["title"], logger=self.logger, api_url=component_data["api_url"], + method=component_data["method"], caption=component_data.get("caption"), headers=component_data.get("headers"), params=component_data.get("params"), + request_body=parsed_body, ) def _create_chatbot_component(self, component_data: dict) -> r.ChatBot: @@ -588,7 +599,7 @@ def _create_chatbot_component(self, component_data: dict) -> r.ChatBot: title=component_data["title"], logger=self.logger, api_url=component_data["api_url"], - model=component_data["model"], + model=component_data.get("model"), caption=component_data.get("caption"), headers=component_data.get("headers"), params=component_data.get("params"), diff --git a/src/vuegen/report.py b/src/vuegen/report.py index 68312aa..147ec14 100644 --- a/src/vuegen/report.py +++ b/src/vuegen/report.py @@ -530,10 +530,14 @@ class APICall(Component): ---------- api_url : str The URL of the API to interact with. + method : str + HTTP method to use for the request ("GET", "POST", or "PUT"). The deafult is "GET". headers : Optional[dict] Headers to include in the API request (default is None). params : Optional[dict] Query parameters to include in the API request (default is None). + request_body : Optional[dict] + The request body for methods like POST or PUT (default is None). """ def __init__( @@ -541,9 +545,11 @@ def __init__( title: str, logger: logging.Logger, api_url: str, + method: str = "GET", caption: str = None, headers: Optional[dict] = None, params: Optional[dict] = None, + request_body: Optional[dict] = None, ): super().__init__( title=title, @@ -552,35 +558,52 @@ def __init__( caption=caption, ) self.api_url = api_url + self.method = method.upper() self.headers = headers or {} self.params = params or {} + # NOTE: request_body is usually dynamically set before the call for POST/PUT + # but we'll include it here if needed for values from a config file + self.request_body = request_body or {} def make_api_request( - self, method: str, request_body: Optional[dict] = None + self, dynamic_request_body: Optional[dict] = None ) -> Optional[dict]: """ Sends an HTTP request to the specified API and returns the JSON response. + It allows overriding the request body dynamically. Parameters ---------- - method : str - HTTP method to use for the request. - request_body : Optional[dict], optional - The request body for POST or PUT methods (default is None). + dynamic_request_body : Optional[dict] + A dictionary to use as the JSON request body for this specific call. + Overrides the instance's request_body if provided. Returns ------- response : Optional[dict] The JSON response from the API, or None if the request fails. """ + request_body_to_send = ( + dynamic_request_body + if dynamic_request_body is not None + else self.request_body + ) try: - self.logger.info(f"Making {method} request to API: {self.api_url}") + self.logger.info(f"Making {self.method} request to API: {self.api_url}") + self.logger.debug(f"Headers: {self.headers}") + self.logger.debug(f"Params: {self.params}") + response = requests.request( - method, + self.method, self.api_url, headers=self.headers, params=self.params, - json=request_body, + # Validate the request body based on the method + json=( + request_body_to_send + if self.method in ["POST", "PUT", "PATCH"] and request_body_to_send + else None + ), ) response.raise_for_status() self.logger.info( @@ -599,10 +622,10 @@ class ChatBot(Component): Attributes ---------- - model : str - The language model to use for the chatbot. api_call : APICall An instance of the APICall class used to interact with the API for fetching chatbot responses. + model : Optional[str] + The language model to use for the chatbot (default is None). headers : Optional[dict] Headers to include in the API request (default is None). params : Optional[dict] @@ -614,8 +637,8 @@ def __init__( title: str, logger: logging.Logger, api_url: str, - model: str, caption: str = None, + model: Optional[str] = None, headers: Optional[dict] = None, params: Optional[dict] = None, ): @@ -630,7 +653,8 @@ def __init__( title=title, logger=logger, api_url=api_url, - caption=caption, + method="POST", + caption=None, headers=headers, params=params, ) diff --git a/src/vuegen/streamlit_reportview.py b/src/vuegen/streamlit_reportview.py index 40ce2ce..534a537 100644 --- a/src/vuegen/streamlit_reportview.py +++ b/src/vuegen/streamlit_reportview.py @@ -256,7 +256,7 @@ def _format_text( """ if type == "header": tag = f"h{level}" - elif type == "paragraph": + elif type == "paragraph" or type == "caption": tag = "p" return f"""st.markdown('''<{tag} style='text-align: {text_align}; color: {color};'>{text}''', unsafe_allow_html=True)""" @@ -417,6 +417,9 @@ def _generate_subsection( subsection_content = [] subsection_imports = [] + # Track if there's a Chatbot component in this subsection + has_chatbot = False + # Add subsection header and description subsection_content.append( self._format_text( @@ -451,15 +454,17 @@ def _generate_subsection( elif component.component_type == r.ComponentType.APICALL: subsection_content.extend(self._generate_apicall_content(component)) elif component.component_type == r.ComponentType.CHATBOT: + has_chatbot = True subsection_content.extend(self._generate_chatbot_content(component)) else: self.report.logger.warning( f"Unsupported component type '{component.component_type}' in subsection: {subsection.title}" ) - # Define the footer variable and add it to the home page content - subsection_content.append("footer = '''" + generate_footer() + "'''\n") - subsection_content.append("st.markdown(footer, unsafe_allow_html=True)\n") + if not has_chatbot: + # Define the footer variable and add it to the home page content + subsection_content.append("footer = '''" + generate_footer() + "'''\n") + subsection_content.append("st.markdown(footer, unsafe_allow_html=True)\n") self.report.logger.info( f"Generated content and imports for subsection: '{subsection.title}'" @@ -753,7 +758,7 @@ def _generate_markdown_content(self, markdown) -> List[str]: def _generate_html_content(self, html) -> List[str]: """ - Generate content for an HTML component in a Streamlit app. + Generate content for an HTML component. Parameters ---------- @@ -813,7 +818,8 @@ def _generate_html_content(self, html) -> List[str]: def _generate_apicall_content(self, apicall) -> List[str]: """ - Generate content for a Markdown component. + Generate content for an API component. This method handles the API call and formats + the response for display in the Streamlit app. Parameters ---------- @@ -834,7 +840,7 @@ def _generate_apicall_content(self, apicall) -> List[str]: ) ) try: - apicall_response = apicall.make_api_request(method="GET") + apicall_response = apicall.make_api_request() apicall_content.append(f"""st.write({apicall_response})\n""") except Exception as e: self.report.logger.error( @@ -851,18 +857,29 @@ def _generate_apicall_content(self, apicall) -> List[str]: ) self.report.logger.info( - f"Successfully generated content for APICall: '{apicall.title}'" + f"Successfully generated content for APICall '{apicall.title}' using method '{apicall.method}'" ) return apicall_content def _generate_chatbot_content(self, chatbot) -> List[str]: """ - Generate content for a ChatBot component. + Generate content to render a ChatBot component, supporting standard and Ollama-style streaming APIs. + + This method builds and returns a list of strings, which are later executed to create the chatbot + interface in a Streamlit app. It includes user input handling, API interaction logic, response parsing, + and conditional rendering of text, source links, and HTML subgraphs. + + The function distinguishes between two chatbot modes: + - **Ollama-style streaming API**: Identified by the presence of `chatbot.model`. Uses streaming + JSON chunks from the server to simulate a real-time response. + - **Standard API**: Assumes a simple POST request with a prompt and a full JSON response with text, + and other fields like links, HTML graphs, etc. Parameters ---------- chatbot : ChatBot - The ChatBot component to generate content for. + The ChatBot component to generate content for, containing configuration such as title, model, + API endpoint, headers, and caption. Returns ------- @@ -871,16 +888,52 @@ def _generate_chatbot_content(self, chatbot) -> List[str]: """ chatbot_content = [] - # Add title + # Add chatbot title as header chatbot_content.append( self._format_text( text=chatbot.title, type="header", level=4, color="#2b8cbe" ) ) - # Chatbot logic for embedding in the web application - chatbot_content.append( - f""" + # --- Shared code blocks (as strings) --- + init_messages_block = """ +# Init session state +if 'messages' not in st.session_state: + st.session_state['messages'] = [] + """ + + render_messages_block = """ +# Display chat history +for message in st.session_state['messages']: + with st.chat_message(message['role']): + content = message['content'] + if isinstance(content, dict): + st.markdown(content.get('text', ''), unsafe_allow_html=True) + if 'links' in content: + st.markdown("**Sources:**") + for link in content['links']: + st.markdown(f"- [{link}]({link})") + if 'subgraph_pyvis' in content: + st.components.v1.html(content['subgraph_pyvis'], height=600) + else: + st.write(content) + """ + + handle_prompt_block = """ +# Capture and append new user prompt +if prompt := st.chat_input("Enter your prompt here:"): + st.session_state.messages.append({"role": "user", "content": prompt}) + with st.chat_message("user"): + st.write(prompt) + """ + + if chatbot.model: + # --- Ollama-style streaming chatbot --- + chatbot_content.append( + f""" +{init_messages_block} + +# Function to send prompt to Ollama API def generate_query(messages): response = requests.post( "{chatbot.api_call.api_url}", @@ -889,6 +942,7 @@ def generate_query(messages): response.raise_for_status() return response +# Parse streaming response from Ollama def parse_api_response(response): try: output = "" @@ -902,43 +956,82 @@ def parse_api_response(response): except Exception as e: return {{"role": "assistant", "content": f"Error while processing API response: {{str(e)}}"}} +# Simulated typing effect for responses def response_generator(msg_content): for word in msg_content.split(): yield word + " " time.sleep(0.1) yield "\\n" -# Chatbot interaction in the app -if 'messages' not in st.session_state: - st.session_state['messages'] = [] +{render_messages_block} -# Display chat history -for message in st.session_state['messages']: - with st.chat_message(message['role']): - st.write(message['content']) +{handle_prompt_block} -# Handle new input from the user -if prompt := st.chat_input("Enter your prompt here:"): - # Add user's question to the session state - st.session_state.messages.append({{"role": "user", "content": prompt}}) - with st.chat_message("user"): - st.write(prompt) - # Retrieve question and generate answer combined = "\\n".join(msg["content"] for msg in st.session_state.messages if msg["role"] == "user") messages = [{{"role": "user", "content": combined}}] with st.spinner('Generating answer...'): response = generate_query(messages) parsed_response = parse_api_response(response) - + # Add the assistant's response to the session state and display it st.session_state.messages.append(parsed_response) with st.chat_message("assistant"): st.write_stream(response_generator(parsed_response["content"])) - """ + """ + ) + else: + # --- Standard (non-streaming) API chatbot --- + chatbot_content.append( + f""" +{init_messages_block} + +# Function to send prompt to standard API +def generate_query(prompt): + try: + response = requests.post( + "{chatbot.api_call.api_url}", + json={{"prompt": prompt}}, + headers={chatbot.api_call.headers} ) + response.raise_for_status() + return response.json() + except requests.exceptions.RequestException as e: + st.error(f"API request failed: {{str(e)}}") + if hasattr(e, 'response') and e.response: + try: + error_details = e.response.json() + st.error(f"Error details: {{error_details}}") + except ValueError: + st.error(f"Response text: {{e.response.text}}") + return None + +{render_messages_block} + +{handle_prompt_block} + + with st.spinner('Generating answer...'): + response = generate_query(prompt) + + if response: + # Append and display assistant response + st.session_state.messages.append({{ + "role": "assistant", + "content": response + }}) + with st.chat_message("assistant"): + st.markdown(response.get('text', ''), unsafe_allow_html=True) + if 'links' in response: + st.markdown("**Sources:**") + for link in response['links']: + st.markdown(f"- [{{link}}]({{link}})") + if 'subgraph_pyvis' in response: + st.components.v1.html(response['subgraph_pyvis'], height=600) + else: + st.error("Failed to get response from API") + """ + ) - # Add caption if available if chatbot.caption: chatbot_content.append( self._format_text( @@ -946,9 +1039,6 @@ def response_generator(msg_content): ) ) - self.report.logger.info( - f"Successfully generated content for ChatBot: '{chatbot.title}'" - ) return chatbot_content def _generate_component_imports(self, component: r.Component) -> List[str]: