Fix for django 1.8+ (parsing args) and files in utf8 #17
Conversation
| self.stdout.write('Working on {}'.format(data_file)) | ||
| if zipfile.is_zipfile(data_file): | ||
| zfile = zipfile.ZipFile(data_file) | ||
| fname = zfile.namelist()[0] | ||
| with zfile.open(fname) as xml_file: | ||
| update_database(xml_file, fname, force_ins) | ||
| else: | ||
| with open(data_file) as xml_file: | ||
| with open(data_file, encoding="utf8") as xml_file: |
There was a problem hiding this comment.
@rexopl on Python 2.7 this throws error that encoding is not known, from what I see encoding was added to io.open in Python 3, so this has to be removed for Python 2.7.
More info I've found on SO https://stackoverflow.com/questions/10971033/backporting-python-3-openencoding-utf-8-to-python-2
There was a problem hiding this comment.
Yeah I was testing on python3 :). Right!
There was a problem hiding this comment.
@rexopl still there are some problems with this and I'm confused why it's saying Unknown filename.
(PanelDjango) [Greyer@Asgaard] 23:05:00
(devel) ~/Documents/Repos/panel-django > ./manage.py teryt_parse addresses/xml/WMRODZ_2018-12-14.xml addresses/xml/TERC_Urzedowy_2018-12-14.xml addresses/xml/SIMC_Urzedowy_2018-12-14.xml addresses/xml/ULIC_Urzedowy_2018-12-14.xml
Working on addresses/xml/WMRODZ_2018-12-14.xml
CommandError: Unknown filename: 'WMRODZ_2018-12-14.xml'
(PanelDjango) [Greyer@Asgaard] 23:05:05
(devel) ~/Documents/Repos/panel-django > ll addresses/xml/
total 220320
drwxr-xr-x 6 Greyer staff 192B Dec 14 23:31 .
drwxr-xr-x 14 Greyer staff 448B Dec 15 18:33 ..
-rw-r--r-- 1 Greyer staff 28M Dec 14 00:00 SIMC_Urzedowy_2018-12-14.xml
-rw-r--r-- 1 Greyer staff 896K Dec 14 00:00 TERC_Urzedowy_2018-12-14.xml
-rw-r--r-- 1 Greyer staff 79M Dec 14 00:00 ULIC_Urzedowy_2018-12-14.xml
-rw-r--r-- 1 Greyer staff 1.5K Dec 14 22:01 WMRODZ_2018-12-14.xml
(PanelDjango) [Greyer@Asgaard] 23:05:37
(devel) ~/Documents/Repos/panel-django >
There was a problem hiding this comment.
There is some problem with new files from GUS. There are even some warnings during zip extractions. I will check it in free time.
There was a problem hiding this comment.
But generally you need to change names to hardcoded names:
fn_dict = OrderedDict([
('WMRODZ.xml', RodzajMiejscowosci),
('TERC.xml', JednostkaAdministracyjna),
('SIMC.xml', Miejscowosc),
('ULIC.xml', Ulica),
])
There was a problem hiding this comment.
So I should invoke
./manage.py teryt_parse addresses/xml/
without filenames?
There was a problem hiding this comment.
Path with filename: ./manage.py teryt_parse addresses/xml/WMRODZ.xml
Please verify, I tested this code with django 1.10.
Note: teryt can work also with djnago 2+ but it needs "cosmetic" changes in models/migrations like adding on_delete, etc.