Lab 3 #3

ValeZh · 2023-08-03T08:26:09Z

I make all

Maybe I don`t do it in time

…trange and the program seems to freeze)

…s maybe works

vloooo · 2023-08-03T21:41:22Z

lab 3/main.py

+logging.debug('Debug message')
+logging.info('Info message')
+logging.warning('Warning message')
+logging.error('Error message')
+logging.critical('Critical message')
+logging.info('1 step')


unnecessary code

vloooo · 2023-08-03T21:41:46Z

lab 3/main.py

+        self.file_out_name = des_fold_path + "\\" + f_name + '.csv'
+        self.data = self.download_data()
+        print(self.file_out_name)
+        logging.info('Made class : set 	Destination folder and  Filename  ')


don't use standard logger. make a custom one

vloooo · 2023-08-03T21:42:04Z

lab 3/main.py

+    def __init__(self, des_fold_path, f_name='output.csv'):
+        self.data_file = 'lab3.csv'
+        self.des_fold_path = des_fold_path
+        self.previous_output = f_name + '.csv'


vloooo · 2023-08-03T21:42:28Z

lab 3/main.py

+        self.data_file = 'lab3.csv'
+        self.des_fold_path = des_fold_path
+        self.previous_output = f_name + '.csv'
+        self.file_out_name = des_fold_path + "\\" + f_name + '.csv'


os.path.join
don't use double-quotes

vloooo · 2023-08-03T21:43:55Z

lab 3/main.py

+        self.previous_output = f_name + '.csv'
+        self.file_out_name = des_fold_path + "\\" + f_name + '.csv'
+        self.data = self.download_data()
+        print(self.file_out_name)


you don't need prints in the code

vloooo · 2023-08-03T21:44:44Z

lab 3/main.py

+    def download_data(self):
+        logging.info('1. Downloading data')
+        respons = requests.get('https://randomuser.me/api/?results=5000&format=csv')
+        open(self.data_file, mode='w', encoding='utf-8').write(respons.text)


you didn't close write stream!!!
use context manager here

vloooo · 2023-08-03T21:45:27Z

lab 3/main.py

+        data_dicts = []
+        with open(self.data_file, 'r', encoding='utf-8') as file:
+            csv_reader = csv.DictReader(file)
+            for row in csv_reader:
+                data_dicts.append(row)


Suggested change

data_dicts = []

with open(self.data_file, 'r', encoding='utf-8') as file:

csv_reader = csv.DictReader(file)

for row in csv_reader:

data_dicts.append(row)

with open(self.data_file, 'r', encoding='utf-8') as file:

data_dicts = list(csv.DictReader(file))

vloooo · 2023-08-03T21:46:02Z

lab 3/main.py

+    def add_to_output_file(self, save_output):
+        with open(self.previous_output, 'w', encoding='utf-8') as csv_output:
+            writer = csv.DictWriter(csv_output, fieldnames=save_output[0].keys())
+            writer.writeheader()
+            writer.writerows(save_output)
+            logging.info('add_to_output_file correct')


the method looks nice

vloooo · 2023-08-19T08:04:46Z

lab3/src/data_humans.py

+logging.basicConfig(
+    filename="../file.log",
+    level=logging.INFO,
+    format="%(asctime)s:%(levelname)s:%(name)s:%(message)s",
+)


don't use basic logger.
make a custom one

vloooo · 2023-08-19T08:04:56Z

lab3/src/data_humans.py

+import requests
+
+logging.basicConfig(
+    filename="../file.log",


don't use double quotes

vloooo · 2023-08-19T08:11:43Z

lab3/src/data_humans.py

+
+class DataHumans:
+    def __init__(self, des_fold_path, f_name="output.csv"):
+        self.data_file = "../lab3.csv"


it's a constant. make it as a class atribute

vloooo · 2023-08-19T08:11:55Z

lab3/src/data_humans.py

+    def __init__(self, des_fold_path, f_name="output.csv"):
+        self.data_file = "../lab3.csv"
+        self.des_fold_path = des_fold_path
+        self.previous_output = f_name + ".csv"


vloooo · 2023-08-19T08:12:30Z

lab3/src/data_humans.py

+        self.data_file = "../lab3.csv"
+        self.des_fold_path = des_fold_path
+        self.previous_output = f_name + ".csv"
+        self.file_out_name = os.path.join(des_fold_path, f"{f_name}.csv")


why don't you use self.previous_output here

vloooo · 2023-08-19T08:13:33Z

lab3/src/data_humans.py

+
+    def download_data(self):
+        logging.info("1. Downloading data")
+        respons = requests.get("https://randomuser.me/api/?results=5000&format=csv")


make this url as a constant

vloooo · 2023-08-19T08:13:52Z

lab3/src/data_humans.py

+    def download_data(self):
+        logging.info("1. Downloading data")
+        respons = requests.get("https://randomuser.me/api/?results=5000&format=csv")
+        with open(self.data_file, mode="w", encoding="utf-8") as file:


make utf-8 another constant

vloooo · 2023-08-19T08:15:12Z

lab3/src/data_humans.py

+            csv_reader = csv.DictReader(file)
+        return [row for row in csv_reader]


Suggested change

csv_reader = csv.DictReader(file)

return [row for row in csv_reader]

return list(csv.DictReader(file))

vloooo · 2023-08-19T08:15:43Z

lab3/src/data_humans.py

+        with open(self.data_file, "r", encoding="utf-8") as file:
+            data_dicts = list(csv.DictReader(file))
+        logging.info(f"downloading correct ,len of file{len(data_dicts)}")
+        return data_dicts


why don't you use read_data_from_file method

vloooo · 2023-08-19T08:16:37Z

lab3/src/data_humans.py

+            csv_reader = csv.DictReader(file)
+        return [row for row in csv_reader]
+
+    def numbering_rows(self, idx, value):


name of functions and methods should be verbs/actions not nouns

vloooo · 2023-08-19T08:18:01Z

lab3/src/data_humans.py

+    def file_replace(self):
+        logging.info("7. Move initial file to the destination folder")
+        if not os.path.exists(self.file_out_name):
+            os.makedirs(self.des_fold_path, mode=777)


google exist_ok=true parameter

vloooo · 2023-08-19T08:19:08Z

lab3/src/data_humans.py

+        return user
+
+    def the_most_old(self, user):  # использовать max lst
+        return max([int(f["dob.age"]) for f in user])


why did you call variable f

vloooo · 2023-08-19T08:20:09Z

lab3/src/data_humans.py

+        return collections.Counter(lst_for_count).most_common(1)[0][0]
+
+    def make_name_of_file(self, basic_path, user_data):
+        return f'{basic_path}\\max_age_{str(self.the_most_old(user_data))}_avg_registered_{str(self.average_year_of_reg(user_data))}_ popular_id_{str(self.popular_genres(user_data))}_user_data_{str(user_data[0]["global_index"])}.csv'


please check if you really need conversion to str

vloooo · 2023-08-19T08:21:37Z

lab3/src/data_humans.py

+        logging.info("9. 10 . make dir for decade")
+        for i in user_data:
+            for c in user_data[i]:
+                basic_path = f"{self.des_fold_path}\\{i}\\{c}"
+                os.makedirs(basic_path)  # f-string \\ make variable//make func
+                with open(
+                    self.make_name_of_file(basic_path, user_data[i][c]),
+                    "w",
+                    encoding="utf-8",
+                ) as csv_output:
+                    writer = csv.DictWriter(
+                        csv_output, fieldnames=user_data[i][c][0].keys()
+                    )
+                    writer.writeheader()
+                    writer.writerows(user_data[i][c])


looks unreadable. let's try to refactor it together

vloooo · 2023-08-19T08:23:11Z

lab3/src/main.py

+if args.numb_of_rows_filt is True:
+    output_file.delete_rows_for_filt(args.numb_of_rows_filt)
+else:
+    output_file.filter_by_gender(args.gender_filt)


try to use ternary operator here

vloooo · 2023-08-19T08:23:46Z

lab3/src/main.py

+pdb.set_trace()
+
+output_file = DataHumans(args.destination_folder, args.file_name)
+if args.numb_of_rows_filt is True:


you don't need to compare var wtih True.
do this:

Suggested change

if args.numb_of_rows_filt is True:

if args.numb_of_rows_filt:

vloooo · 2023-08-19T08:24:06Z

lab3/src/main.py

+parser.add_argument("--file_name", type=str, required=True)
+parser.add_argument("-g", "--gender_filt", type=str)
+parser.add_argument("-n", "--numb_of_rows_filt", type=int)
+parser.add_argument("--log_level", type=str)


the parameter should be positional

vloooo · 2023-08-19T08:24:30Z

lab3/src/main.py

+parser.add_argument("-g", "--gender_filt", type=str)
+parser.add_argument("-n", "--numb_of_rows_filt", type=int)


they should be mutually exclusive

vloooo · 2023-08-19T08:24:58Z

lab3/src/main.py

+# использовать argparse
+parser = argparse.ArgumentParser()
+parser.add_argument("--destination_folder", type=str, required=True)
+parser.add_argument("--file_name", type=str, required=True)


the parameter must have default value. not required

vloooo · 2023-08-19T08:26:04Z

lab3/tests/unit/test_lab3_main.py

+#     with patch("builtins.open", mock_open(read_data="data")) as mock_file:
+#         res = dh.read_data_from_file()
+#         mock_file.assert_called_with(dh.previous_output, "r", encoding="utf-8")
+#         assert res == ['test']


add blank line at the end of the file

ValeZh added 9 commits July 23, 2023 21:34

I started doing it

374b8c9

Maybe I don`t do it in time

i make 3 tasks

9f14698

don`t worry

860660c

I make all tasks without loging (only the packing of the archive is s…

1a4acbc

…trange and the program seems to freeze)

This version have strange bug with name of csv file but another thing…

a6418fa

…s maybe works

Commit before putting in order

f7d6dfe

I tried to prettify

57854b9

I make all

383cfff

I tried there to make test

2cda3a6

vloooo reviewed Aug 3, 2023

View reviewed changes

ValeZh and others added 5 commits August 5, 2023 16:26

I make all

b28038b

I make all

c6a94c9

I make all but something strange with argparse

f39f4c0

I tried to make test it`s very hard

0107a1e

Delete lab4 directory

983ed16

vloooo reviewed Aug 19, 2023

View reviewed changes

lab3/src/data_humans.py

import requests

logging.basicConfig(

filename="../file.log",

Copy link

Collaborator

vloooo Aug 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use double quotes

vloooo reviewed Aug 19, 2023

View reviewed changes

ValeZh added 3 commits August 20, 2023 19:51

some changes

3ad6a70

Merge branch 'lab_3' of https://github.com/ValeZh/pythonrep into lab_3

961f3b7

some changes

c1c2e94

		csv_reader = csv.DictReader(file)
		return [row for row in csv_reader]

	csv_reader = csv.DictReader(file)
	return [row for row in csv_reader]
	return list(csv.DictReader(file))

	if args.numb_of_rows_filt is True:
	if args.numb_of_rows_filt:

		parser.add_argument("-g", "--gender_filt", type=str)
		parser.add_argument("-n", "--numb_of_rows_filt", type=int)

Lab 3 #3

Are you sure you want to change the base?

Lab 3 #3

Uh oh!

Conversation

ValeZh commented Aug 3, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants