WIP : added support for century #821

NEERAJAP2001 · 2020-10-27T12:23:44Z

@noviluni sir, This is PR which show's the code for error #817

This PR add's support for century word #725
So, here I was unable to generate en.py because of this (#817 ) error !!
Thanks

noviluni · 2020-10-27T13:50:37Z

Hi @NEERAJAP2001,

to generate the files you should do (from the dateparser directory):
pip install -r dateparser_scripts/requirements.txt

and then:
python dateparser_scripts/write_complete_data.py

I tried with the code you uploaded and it worked, could you try to follow this and tell me if you encounter any issue?

thanks in advance

NEERAJAP2001 · 2020-10-27T14:41:02Z

Okay, sir, I will let you know if I have any problems
Thanks!!

NEERAJAP2001 · 2020-10-28T11:46:50Z

I ran this :

pip install -r dateparser_scripts/requirements.txt

It shows :

Requirement already satisfied: gitpython in c:\python38\lib\site-packages (from -r dateparser_scripts/requirements.txt (line 1)) (3.1.9)
Collecting orderedset
  Using cached orderedset-2.0.3.tar.gz (101 kB)
Requirement already satisfied: ruamel.yaml in c:\python38\lib\site-packages (from -r dateparser_scripts/requirements.txt (line 3)) (0.16.12)
Requirement already satisfied: gitdb<5,>=4.0.1 in c:\python38\lib\site-packages (from gitpython->-r dateparser_scripts/requirements.txt (line 1)) (4.0.5)
Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in c:\python38\lib\site-packages (from ruamel.yaml->-r dateparser_scripts/requirements.txt (line 3)) (0.2.2)
Requirement already satisfied: smmap<4,>=3.0.1 in c:\python38\lib\site-packages (from gitdb<5,>=4.0.1->gitpython->-r dateparser_scripts/requirements.txt (line 1)) (3.0.4)
Building wheels for collected packages: orderedset
  Building wheel for orderedset (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-wrb0sshz\\orderedset\\setup.py'"'"'; __file__='"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-wrb0sshz\\orderedset\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\neeraj\AppData\Local\Temp\pip-wheel-wn8jr925'
       cwd: C:\Users\neeraj\AppData\Local\Temp\pip-install-wrb0sshz\orderedset\
  Complete output (17 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win32-3.8
  creating build\lib.win32-3.8\orderedset
  copying lib\orderedset\__init__.py -> build\lib.win32-3.8\orderedset
  running build_ext
  building 'orderedset._orderedset' extension
  creating build\temp.win32-3.8
  creating build\temp.win32-3.8\Release
  creating build\temp.win32-3.8\Release\lib
  creating build\temp.win32-3.8\Release\lib\orderedset
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x86\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\python38\include -Ic:\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include" /Tclib/orderedset/_orderedset.c /Fobuild\temp.win32-3.8\Release\lib/orderedset/_orderedset.obj
  _orderedset.c
  c:\python38\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x86\\cl.exe' failed with exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for orderedset
  Running setup.py clean for orderedset
Failed to build orderedset
Installing collected packages: orderedset
    Running setup.py install for orderedset ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-wrb0sshz\\orderedset\\setup.py'"'"'; __file__='"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-wrb0sshz\\orderedset\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\neeraj\AppData\Local\Temp\pip-record-nocggnmh\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python38\Include\orderedset'
         cwd: C:\Users\neeraj\AppData\Local\Temp\pip-install-wrb0sshz\orderedset\
    Complete output (17 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win32-3.8
    creating build\lib.win32-3.8\orderedset
    copying lib\orderedset\__init__.py -> build\lib.win32-3.8\orderedset
    running build_ext
    building 'orderedset._orderedset' extension
    creating build\temp.win32-3.8
    creating build\temp.win32-3.8\Release
    creating build\temp.win32-3.8\Release\lib
    creating build\temp.win32-3.8\Release\lib\orderedset
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x86\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\python38\include -Ic:\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include" /Tclib/orderedset/_orderedset.c /Fobuild\temp.win32-3.8\Release\lib/orderedset/_orderedset.obj
    _orderedset.c
    c:\python38\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x86\\cl.exe' failed with exit status 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-wrb0sshz\\orderedset\\setup.py'"'"'; __file__='"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-wrb0sshz\\orderedset\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\neeraj\AppData\Local\Temp\pip-record-nocggnmh\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python38\Include\orderedset' Check the logs for full command output.

I tried pip install orderset but it again showed :

pip install orderedset
Collecting orderedset
  Using cached orderedset-2.0.3.tar.gz (101 kB)
Building wheels for collected packages: orderedset
  Building wheel for orderedset (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-hhqapt4x\\orderedset\\setup.py'"'"'; __file__='"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-hhqapt4x\\orderedset\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\neeraj\AppData\Local\Temp\pip-wheel-lvbpnher'
       cwd: C:\Users\neeraj\AppData\Local\Temp\pip-install-hhqapt4x\orderedset\
  Complete output (17 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win32-3.8
  creating build\lib.win32-3.8\orderedset
  copying lib\orderedset\__init__.py -> build\lib.win32-3.8\orderedset
  running build_ext
  building 'orderedset._orderedset' extension
  creating build\temp.win32-3.8
  creating build\temp.win32-3.8\Release
  creating build\temp.win32-3.8\Release\lib
  creating build\temp.win32-3.8\Release\lib\orderedset
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x86\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\python38\include -Ic:\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include" /Tclib/orderedset/_orderedset.c /Fobuild\temp.win32-3.8\Release\lib/orderedset/_orderedset.obj
  _orderedset.c
  c:\python38\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x86\\cl.exe' failed with exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for orderedset
  Running setup.py clean for orderedset
Failed to build orderedset
Installing collected packages: orderedset
    Running setup.py install for orderedset ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-hhqapt4x\\orderedset\\setup.py'"'"'; __file__='"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-hhqapt4x\\orderedset\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\neeraj\AppData\Local\Temp\pip-record-jfws0gt7\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python38\Include\orderedset'
         cwd: C:\Users\neeraj\AppData\Local\Temp\pip-install-hhqapt4x\orderedset\
    Complete output (17 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win32-3.8
    creating build\lib.win32-3.8\orderedset
    copying lib\orderedset\__init__.py -> build\lib.win32-3.8\orderedset
    running build_ext
    building 'orderedset._orderedset' extension
    creating build\temp.win32-3.8
    creating build\temp.win32-3.8\Release
    creating build\temp.win32-3.8\Release\lib
    creating build\temp.win32-3.8\Release\lib\orderedset
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\bin\HostX86\x86\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\python38\include -Ic:\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.27.29110\include" /Tclib/orderedset/_orderedset.c /Fobuild\temp.win32-3.8\Release\lib/orderedset/_orderedset.obj
    _orderedset.c
    c:\python38\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x86\\cl.exe' failed with exit status 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-hhqapt4x\\orderedset\\setup.py'"'"'; __file__='"'"'C:\\Users\\neeraj\\AppData\\Local\\Temp\\pip-install-hhqapt4x\\orderedset\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\neeraj\AppData\Local\Temp\pip-record-jfws0gt7\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python38\Include\orderedset' Check the logs for full command output.

I even tried :

pip install pip --upgrade
pip install --upgrade setuptools

But still, It shows the same error.
I even downloaded vs code installer for windows 10 for C++ build tools!

Gallaecio · 2020-10-30T12:35:49Z

You probably need to install the Visual C++ Build Tools (a free Windows package) to be able to build certain Python packages in Windows.

NEERAJAP2001 · 2020-11-05T08:16:17Z

Yes, sir, I have downloaded it again!!
it still show's the same error!!

noviluni · 2020-11-05T09:54:20Z

Hi @NEERAJAP2001, sorry for the late answer. Your error says that there is an error when trying to install orderedset, however, I removed the orderedset dependency as part of this PR (#825), could you update your code?

You can do it by pulling from master and merge or rebase against it. In that way you won't need orderedset and you shouldn't have any issue. Let me know if it works or if you need help.

NEERAJAP2001 · 2020-11-05T12:25:40Z

So, @noviluni sir should I re-code (delete the existing code for century support and code again) and generate the en.py now?

kishan3 · 2020-11-06T06:00:39Z

You can do it by pulling from master

@NEERAJAP2001 you can try following in your forked repo.

git remote add upstream https://github.com/scrapinghub/dateparser.git
git pull upstream master

and then try to install requirements.

Reference: https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/configuring-a-remote-for-a-fork

NEERAJAP2001 · 2020-11-06T10:08:11Z

I'll try that thanks!!

NEERAJAP2001 · 2020-11-06T10:40:40Z

Yes, @kishan3, sir, It worked. I was able to download all the requirements.
Thanks!!
#821 (comment) as mentioned here, I ran both commands
But after running this python dateparser_scripts/write_complete_data.py command to generate en.py threw this error.

Traceback (most recent call last):
  File "dateparser_scripts/write_complete_data.py", line 113, in <module>
    with open(cldr_numeral_directory + language + '.json') as f:
  File "dateparser_scripts/write_complete_data.py", line 93, in write_complete_data

  File "dateparser_scripts/write_complete_data.py", line 54, in _get_complete_date_translation_data
    cldr_data = {}
  File "C:\Python38\lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
  File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 1039: character maps to <undefined>

as same as the #817 issue!!

noviluni · 2020-11-06T10:46:05Z

Hi @NEERAJAP2001, that's weird... 🤔

Could you try changing this line:

with open(cldr_date_directory + language + '.json') as f:

to this:

with open(cldr_date_directory + language + '.json', encoding='utf-8') as f:

(line 53 in write_complete_data.py)

And try again?

NEERAJAP2001 · 2020-11-06T11:16:39Z

It shows this

   with open(cldr_numeral_directory + language + '.json') as f:
 File "dateparser_scripts/write_complete_data.py", line 93, in write_complete_data

 File "dateparser_scripts/write_complete_data.py", line 57, in _get_complete_date_translation_data
   with open(cldr_date_directory + language + '.json') as f:
 File "C:\Python38\lib\site-packages\ruamel\yaml\loader.py", line 69, in __init__
   Reader.__init__(self, stream, loader=self)
 File "C:\Python38\lib\site-packages\ruamel\yaml\reader.py", line 85, in __init__
   self.stream = stream  # type: Any  # as .read is called
 File "C:\Python38\lib\site-packages\ruamel\yaml\reader.py", line 130, in stream
   self.determine_encoding()
 File "C:\Python38\lib\site-packages\ruamel\yaml\reader.py", line 190, in determine_encoding
   self.update_raw()
 File "C:\Python38\lib\site-packages\ruamel\yaml\reader.py", line 297, in update_raw
   data = self.stream.read(size)
 File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
   return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 391: character maps to <undefined>```

noviluni · 2020-11-06T11:21:12Z

@NEERAJAP2001 you should probably add the , encoding='utf-8' parameter to all the places in that file that you find open.

NEERAJAP2001 · 2020-11-06T11:22:08Z

Oh okay!! I'll try and let you know sir!!

added support for century

e961feb

NEERAJAP2001 mentioned this pull request Oct 27, 2020

Unable to generate en.py #817

Closed

NEERAJAP2001 closed this Mar 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP : added support for century #821

WIP : added support for century #821

NEERAJAP2001 commented Oct 27, 2020

noviluni commented Oct 27, 2020

NEERAJAP2001 commented Oct 27, 2020

NEERAJAP2001 commented Oct 28, 2020

Gallaecio commented Oct 30, 2020

NEERAJAP2001 commented Nov 5, 2020

noviluni commented Nov 5, 2020 •

edited

Loading

NEERAJAP2001 commented Nov 5, 2020

kishan3 commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

noviluni commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

noviluni commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

WIP : added support for century #821

WIP : added support for century #821

Conversation

NEERAJAP2001 commented Oct 27, 2020

noviluni commented Oct 27, 2020

NEERAJAP2001 commented Oct 27, 2020

NEERAJAP2001 commented Oct 28, 2020

Gallaecio commented Oct 30, 2020

NEERAJAP2001 commented Nov 5, 2020

noviluni commented Nov 5, 2020 • edited Loading

NEERAJAP2001 commented Nov 5, 2020

kishan3 commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

noviluni commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

noviluni commented Nov 6, 2020

NEERAJAP2001 commented Nov 6, 2020

noviluni commented Nov 5, 2020 •

edited

Loading