myLab Data Manager allows the user to sort, organise and manipulate data from myLab chemical inventory list. The can easily visualize the information using just a few clicks and inputs. After locating the items, the user can also transfer the chemicals to the different worksheets and manage data by evaluating the remaining quantity of the chemicals.
As a chemist, who has seen chemical inventories evolve over the years. It used to be lab records earlier, then arrived excel sheets saved on computers, then it became connected with internet and shared. Nowadays it has evolved to electronic chemical inventories, where one can visualize entire institutes inventories at one place, allowing the lab inventory organisation using auditing trail for every sample. it means there can be easy exchange of information and very smooth sharing mechanism possible. This idea of myLab data manager can be easily extended to make a similar application in combination with excel sheet. Using which one can update the list and organize data without individually going and searching long lists of excel worksheets but just entering a small keyword and the required piece of information would be presented to the user.
-
- As a user of this application I want:
- to be able to see options guiding how to proceed, so that parse the chemical list easily using just selecting options.
- to be able to see the whole list of chemicals and their details enlisted, so that I can make an appropriate idea how many chemicals exist in the main chemical inventory.
- to be able to see the whole list of chemicals and their details enlisted, so that I can make an appropriate idea of which brands and destinations can I look for the chemicals.
- to be provided options to execute keyword inputs based search, so that I don't have to type in the whole complicated chemical name to locate them.
- to be provided exact input based search, so that I don't get a broad search result that needs further time to analyze.
- my search be very specific, so that I can get the result directly without enetering too many details.
- to be able to locate individual chemical details based on keywords search of the chemical/ brand / destination names, or quantities, so that I don't need to remember every detail of the chemical.
- to be able to locate the list of undefined amounts (with missing information) and update them in assess category, so that I can look it up and add details later.
- to be able to locate the fully filled bottles and update them in storage category, so that full bottles can be stored and open ones can be used.
- to be able to locate the empty bottles and update them deleted_items category, so that I can have a record to know which chemicals are over.
- to be able to manually input data and update them manual_entry category, so that I can add the chemicals and their details to record them.
- to be able to shown the all the options until I choose to exit, so that I don't need to start the application over and over again.
As a creator of this application I want:
- to offer the user the list of options to choose from, so that they can proceed with just one input.
- to ask user inputs step by step by asking them questions, so that I can find their answers quickly.
- to be able to help user locate the chemicals with simple keyword search, so that user doesn't need to enter large chemical names to enable search.
- to be able to to help user locate the chemicals with exact search like with chemical names, so that user doesn't need to go through a large list to arrive at the answer.
- to enable user to locate chemicals ith incomplete information and update assess worksheet, so thet user can have all results in one place for futher check.
- to enable user to locate filled bottles and update storage worksheet, so thet they can have the information of new bottles in one place.
- to enable user to locate empty bottles and update deleted_items worksheet, so thet they can have the information of deleted items in one place.
- to enable user to manually enter data and update manual_entry worksheet, so thet they can add in data by typing in manually.
- to show all the options until user chooses to exit, so that they don't need to start the application all over again.
- to give the user a smooth way of parsing and analysing data, so that user doesn't nedd to go to excel sheet and type in names and arrive at single cells but all results in one place.
-
The myLab Data Manager Structure consists of the first section that will be visible to the user consists of three parts (depicted in the following image):
- Greeting to the user along with an introduction to the application.
- Instruction to the user followed by the options for the user to select.
- An input area for user's option selection. The user can select a number that matches the option they want to proceed with.
User will be asked shown the 2nd and 3rd parts prompting user input until the user selects the option 10 (exit).
The flow chart was created using Lucidchart, a web-based diagramming application. The flow charts shown below, shows the design concept of myLab Data Manager application. It shows how upon selecting options, different kinds of lists will appear. Options 1-5 display results within the console.
Upon selecting options 6-9, the results will also appear on the console and also the simultaneous updates will be take place in Google Sheets spreadsheet 'myLab_data'.
-
- Option 1
Upon selection of option 1 or when the user makes an input 1, the whole chem_inventory worksheet gets displayed on the screen. The google sheet being analysed has 2520 entries and the whole list will be displayed on the screen. Next options will allow the user to narrow down the search further.
- Option 2
Upon option 2 selection, user will be asked to make an entry of chemical name they wish to find. When the user makes an entry, all the input containing results will appear on the screen. Following which, the user will be asked a question, if the search was helpful. The result that appears on first input contains many chemicals as the search is not as a result of specific string full match.
If the user selects n (no) and suggest user is not content with the result, User will be asked to enters more specific keyword. Upon even the same input, chemical list containing exactly the typed in keyword will appear on the screen. And a message, “Great! You got it now!!’ If the user selects y (yes), “Awesome!”will be printed. If no entry, “Please enter y/n!” will be printed.
- Option 3
Upon selection of option 3, user will asked to enter the keyword from the destination name. When the user makes the keyword search, all the keyword containing results will appear on the screen. This is not based on full match search. Smaller lists will be fully visible and big lists would appear as shown below (i.e.not full data will be displayed). To guide the user, a message asks the user to enter slightly more specific keyword. Upon increasing specificity, the result will become more apt.
-
Option 4 Upon selection of option 4, user will asked to enter the exact quantity input to search. When the user enters the value, all the input results will appear on the screen, which will be fully matching the input made. This is delibeartely made more specific, because numbers are similar and the list is too long. If user doesn’t enter the exact quantity for the search, empty index will be returned.
-
Option 5 Upon selection of option 5, user will asked to enter the brand name for the search. When the user enters the keyword, all the keyword containing results will appear on the screen.
- Option 10
Upon selection of option 10, user will be given an exit message “See you later then!!”
-
- Option 6
Upon selection of option 6, the data containing undefined values/ empty cells in the chem_inventory main worksheet will be targeted. The list that is retrieved from chemical inventory list, wwill get printed at the terminal and also gets updated in the assess worksheet of myLab_data spreadsheet simultaneously.
- Option 7
Upon selection of option 7, the two columns 'Total Amount' and 'Amount remaining' will be searched and when both the cells contain same values, data is retrieved based on the full bottles. And the values will be displayed at the terminal and also will get updated in the 'storage' worksheet.
- Option 8
Upon selection of option 8, the two column 'Amount remaining' will be searched for 0 g / 0 ml entries depicting empty bottles. Retrieved data from chemical inventory list printed at the terminal and gets updated in the 'deleted_items' worksheet of myLab_data spreadsheet.
- Option 9
Upon selection of option 9, user will asked to add the data for manual entry. The values to be entered will be displayed. Only upon adding 5 values, will the manual_entry table be updated. If the user enters more than 5 values, “invalid input” will be displayed and user will be asked to enter values again.
If the user enters valid input, then the manual_entry worksheet will get updated and a message will appear that it has been updated successfully.
This tool is created purely using Python language.
-
Git - For version control These commands were used for version control during project:
- git add . - To add files before committing
- git commit -m "type your message mentioning changes" - To commit changes to the local repository
- git push - To push all committed changes to the GitHub repository
-
GitHub - To create my repositories, save and store my project files.
-
pandas- Python Data Analysis Library pandas is an open source data analysis and manipulation tool, built on top of the Python programming language. It supports Python 3.8, 3.9 and 3.10 officially. My codes are largely based on pandas code, as they contain extensive functions very well suited for parsing and analysing data.
-
gspread is a Python API for Google Sheets and requires Python 3+. It allows user to open, read, write, share spreadsheets. Additionally it enables user to select, create, delete worksheets and to format cell ranges.
-
pprint — Data pretty printer — Python 2.7, 3.5 onwards. The pprint module provides “pretty-print” to Python data structures. The formatted representation arranges objects on a single line if it can, and breaks them onto multiple lines if they don’t fit within the allowed width. I used it in order to visualize the lists in more user friendly style.
-
Issue
-
One of the major issues I faced was my attempt to enable manual data entry to assess list. In my early design strategy, I wanted that user can also type in input of the chemicals and details and add it to the retrieved list within the function - update_assess_worksheet(). But I couldn't achieve it because of the two lists being different in their types. As my method of retriving the data uses pandas library, the list is extracted as dataframes.
However, the manual data entry a user makes is actually comma separated values and string value needs to be converted into a list of values. Using the split() method, we break up commas and convert into list. The dataframes and the list do not add up well. The manual data (which now appends in a new list named manula_entry) appends to the list but it appears to append from coumn E not column A, as shown in figure below.
I tried various methods to align the lists by coverting the manual entry list to dataframes, but they weren't fruitful. Finally I decided to make a manual_entry worksheet where the entries can be appended in a new list.
-
I faced a problem with an input in display_chem_keyword_search() function. When I needed two inputs in the question for the user:
print('\nDid the search help you?[y/n]\n') i = input() if i == 'n': i = input('\nTry more specific keyword: \n') if i not in df: print('\nDisplaying chemicals and details:\n')
This caused the function to work, but the input needed to be entered 3 or 4 times. For eg. n n n and then the print statement 'Displaying chemicals and details:' would appear.
This was solved with my mentor's help and I learnt this that one must enter i = input('\nDid the search help you?[y/n]\n')
rather than using
print('\nDid the search help you?[y/n]\n') i = input()
and the problem got solved.
-
As advised by tutors that pep8 no longer working, I validated Code Institute Python linter. Test result: No errors found. It showed no warnings or problems as in screenshot below.
myLab Data Manager application was deployed using Heroku - a platform as a service (PaaS) that enables developers to build, run, and operate applications fully in the cloud. The folloing steps were followed for deployment.
- Visit Heroku.com and create a new account.
- Click the 'New' button on the top right corner, and select 'Create new app' from the dropdown menu,
- Create a name for your app using available App names in lower-case letters, numbers and dashes. This app was named "mylab-data-manager"
- Select region, in this case, 'Europe'.
- Click on the 'Create App' button.
- The app gets created in Heroku and Heroku dashboard gets opened.
- Before any action, settings must be defined. Navigate to the settings tab and scroll down to the button 'Reveal Config vars'.
- For the 'KEY', enter 'CREDS' in the input area and then for word 'VALUE', copy and paste the whole data from 'creds.json' file then click on the 'Add' button.
- Scroll down to 'Buildpacks'. Click the 'Add Buildpack' button.
- From the pop up window, select 'python' and save changes.
- Repeat this again but this time selecting 'node.js' and save the changes. It is 'important' to make sure the buildpacks are in the correct order with 'Python' first and 'node.js' second. If they are not in the correct order, drag them into the right order.
- After the settings are done, navigate to the 'Deploy' tab at the top left side.
- Select the option 'Github, 'connect to github' as the deployment method.
- Search for the Github Repository in the search field (in this case 'myData_Data_Manager') and start 'Search'.
- When the search is complete the link to github repository will appear, click 'connect'.
- Once the repository is connected to Heroku, Click the 'Enable Automatic Deploys' button for automatic deployment.
- Alternatively one can manually deploy by selecting a branch to deploy from and clicking 'Deploy Branch'.
- If 'Enable Automatic Deploys' is chosen, Heroku will build a new version of the app when a change to 'gitpod' is pushed to 'Github'.
- Manual deployment allows the update the app whenever 'Deploy Branch' is clicked.
- In case of this project, I selected 'Enable Automatic Deploys' to ensure the code was deployed straight away at each push from 'Gitpod'.
- Once the build process completes (this can take a few seconds), live app could be viewed by clicking button 'View' below 'Your app was successfully deployed'.
- Link generated: https://mylab-data-manager.herokuapp.com/
-
A few references other than https://learn.codeinstitute.net/ci_program/diplomainfullstacksoftwarecommoncurriculum used for coding:
-
Stack overflow
-
To set up while loop user selection: http://introtopython.org/while_input.html
-
For installation of panda library: https://www.codegrepper.com/code-examples/shell/pandas+pip+install
-
Function to set keyword search using display_chem_keyword_search(): https://www.geeksforgeeks.org/select-rows-that-contain-specific-text-using-pandas/
-
pandas.Series.str.contains to search using keyword: https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.Series.str.contains.html
-
Function to check empty cells in a row: https://stackoverflow.com/questions/66888339/find-empty-cells-in-rows-of-a-column-dataframe-pandas
-
Append values to worksheet using pandas: https://medium.com/@jb.ranchana/write-and-append-dataframes-to-google-sheets-in-python-f62479460cf0
-
To concatante dataframes: https://pandas.pydata.org/docs/user_guide/merging.html
-
To update dataframes in worksheet: https://docs.gspread.org/en/latest/user-guide.html
-
To update storage worksheet, two columns containing same value in a row: https://stackoverflow.com/questions/56198775/find-rows-having-same-values-in-multiple-columnsnot-all-columns-in-pandas-data
-
Youtube videos were referred to in order to understand and clarify many operations.
- My Mentor Jubril Akolade for his guidance, best suggestions and constant encouragement. I very much appreciate his coding tips that helped me solve really complicated challenges I faced.
- My fellow students for their company and encouragement. Kenan, Lane and Paul for guiding us through. Special thanks to Kristyna for her encouragement and support.
- My tutors who helped me understand the concepts better.