[PYPI] Turn into front-end API/package #21

.github/workflows/compile_publish.yml

@@ -10,37 +10,68 @@ on:
 
 jobs:
   create-release:
-    name: "Create Release"
+    name: "Preparations"
     runs-on: ubuntu-latest
     outputs:
       upload_url: ${{ steps.release.outputs.upload_url }}
       id: ${{ steps.release.outputs.id }}
+      version: ${{ steps.version.outputs.sv-dlp_version }}
 
     steps:
     - name: Switch Branch
       uses: actions/checkout@v2
       with:
         ref: 'master'
 
-    - name: Generate Version
-      id: ver
-      run: echo "::set-output name=ver::$(sed -e "s#.*=\(\)#\1#;s/'//g" sv-dlp/version.py)"
+    - name: Bump version
+      id: version
+      run: python dev_scripts/update-version.py
 
     - name: Get Changelog
-      run: sed '/## '${{ steps.ver.outputs.ver }}'/d;/'---'/Q' CHANGELOG.md >> changelog_body
+      run: sed '/## '${{ steps.version.outputs.sv-dlp_version }}'/d;/'---'/Q' CHANGELOG.md >> changelog_body
+
+    - name: Push Changes
+      uses: stefanzweifel/git-auto-commit-action@v4
+      with:
+        commit_message: '[${{ steps.version.outputs.sv-dlp_version }}] Bump Version & Changelog'
 
     - name: Create Release
       id: release
       uses: actions/create-release@master
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       with:
-        tag_name: ${{ steps.ver.outputs.ver }}
-        release_name: ${{ steps.ver.outputs.ver }}
+        tag_name: ${{ steps.version.outputs.sv-dlp_version }}
+        release_name: ${{ steps.version.outputs.sv-dlp_version }}
         body_path: ./changelog_body
         draft: true
         prerelease: ${{ github.event.inputs.pre_release }}
 
+  build-publish-pypi:
+    name: "Build & Publish to PyPi"
+    needs: create-release
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Switch Branch
+      uses: actions/checkout@v2
+      with:
+        ref: 'master'
+
+    - name: Set up Python 3.10.5
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.10.5'
+        architecture: 'x64'
+
+    - name: Build Package
+      run: python setup.py sdist bdist_wheel
+
+    - name: Publish to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        password: ${{ secrets.PYPI_API_TOKEN }}
+
   compile-linux:
       name: "Compile Linux"
       runs-on: ubuntu-latest

.gitignore

@@ -6,4 +6,5 @@ __pycache__/
 *.py[cod]
 /build
 /dist
-*.spec
+*.spec
+/sv_dlp.egg-info

CHANGELOG.md

@@ -1,3 +1,18 @@
+## 2023.01.16
+## Changelog
+The functionality of sv-dlp's back-end can now be utilized as a constructor in other scripts, providing developers to 
+easily incorporate the functionality of sv-dlp into their own projects. This makes it possible to access the various 
+features and capabilities of sv-dlp, such as the ease of metadata scrapping between different services,
+panorama download & stitching, etc.
+
+## Added
+- **[SERVICES]** All metadata is now returned in an unique readable format.
+- **[SERVICES]** Historical Panorama & Linked Panoramas now appear in metadata.
+- **[SERVICES]** Service can now be determined from input (`get_available_services`).
+- **[DOWNLOAD]** TQDM Download is now properly formatted.
+## Fixed
+- **[BING]** Fixed Bing not downloading tiles properly
+---
 ## 2022.07.31
 ## Fixed
 - **[APPLE]** Fixed an issue where coordinates were not being converted to Panorama IDs correctly

README.md

@@ -17,44 +17,46 @@ is owned by their respective authors.
 ## **Features**
 - Download Panoramas
     - A panorama ID, a coordinate or a short URL can be parsed
-        - **[GOOGLE]** A can radius can be parsed as well 
-    - Is multithreaded, meaning that each row is being downloaded at the same time enhancing speed
+        - **[Google]** Radius can be parsed if input is coordinates 
+    - Is multithreaded; each panorama tile row is in different threads for faster downloads
     - Saves individual tiles if `--save-tiles` is parsed
-    - Download from .csv/.json input obtained from [Nur's Map Generator](https://map-generator-flax.vercel.app/)
+    - Download from .csv/.json files
 - Fully print metadata
-    - Date and coords can also be printed too separately
-- Coordinates and short URLs are automatically turned to panorama IDs
-- Short links with a panorama ID (or coordinates aswell)
-
-A built-in generator is also planned for intensive tasks that the web browser version cannot perform well due to JavaScript's inefficiency.
+    - Coords, Date, Panorama ID and Gen can be printed separately
+- Coordinates and short URLs are automatically turned to Panorama IDs
+- Short links with a Panorama ID or coordinates
 
 ## **Services**
 Service|Status|Notes
 :------|:-----|:----
 Google |100%  |Zoom 5 only obtains a portion of the face, but that might be Google's problem.
-Baidu  |100%  |Downloading might be slow depending on how far you are from Baidu's servers
-Yandex |99%   |Short Link function does not use the `shortenPath` API call. Only crops panorama for one specific resolution.
-Apple  |99%   |Misc features not supported; can only obtain metadata by coords - credit to [sk-zk & retroplasma](https://github.com/juanpisuribe13/sv-dlp/blob/master/CREDITS)
-Bing   |95%   |Misc features not implemented; can only obtain metadata by coords - credit to [sk-zk](https://github.com/juanpisuribe13/sv-dlp/blob/master/CREDITS)
+Baidu  |99%   |Downloading might be slow depending on how far you are from Baidu's servers; translation between BD09MC and WGS84 is innacurate
+Yandex |99%   |Short Link function does not use the `shortenPath` API call.
+Apple  |95%   |Misc features not supported; can only obtain metadata by coords - credit to [sk-zk & retroplasma](https://github.com/juanpisss/sv-dlp/blob/master/CREDITS)
+Bing   |95%   |Misc features not implemented; can only obtain metadata by coords - credit to [sk-zk](https://github.com/juanpisss/sv-dlp/blob/master/CREDITS)
 
 ## **Installation**
-### **Windows**
-[W.I.P]
-### **Linux**
-[W.I.P]
+### Automatic
+```bash
+$ python3 -m pip install sv_dlp
+# Make sure Python is in PATH
+# if you plan on using its CLI
+```
+### Manual
+If you prefer using a binary instead, check out the binaries [here](https://github.com/juanpisss/sv-dlp/releases/latest)
+and add them to your `PATH` folder. Check out this [guide](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/)
+for more information.
 
 ## **FAQ**
-## **Why aren't all my coordinates from .csv downloading?**
+### **Why aren't all my coordinates from .csv downloading?**
 Depending on how close the given coordinates are, it'll download the same panorama ID no matter the difference. It's recommended to use .json or .csv with panorama IDs instead.
 ## **Can X Service be added?**
 Depending on how accesible it is scraping-wise, yes.
 **Feel free to submit in an issue** about the specific service and I'll gladly see what I can do about it. If you're a developer though, refer to the next question.
-## **I'm a developer and I want to add X service or improve the code. Can I?**
+### **I'm a developer and I want to add X service or improve the code. Can I?**
 Sure! You're welcome to submit in a pull request as long as its to improve
 this program, such as improving performance, fixing a bug, adding a service, etc.
-For those who want to add a service, check out [the documentation for it](https://github.com/juanpisuribe13/sv-dlp/blob/master/extractor/README.md).
-## **What does sv-dlp stand for?**
-Inspired from yt-dlp, sv-dlp is more than a panorama scrapper.
+For those who want to add a service, check out [the documentation for it](https://github.com/juanpisss/sv-dlp/blob/master/extractor/README.md).
 
 ## **License**
-[MIT](https://raw.githubusercontent.com/juanpisuribe13/sv-dlp/master/LICENSE)
+[MIT](https://raw.githubusercontent.com/juanpisss/sv-dlp/master/LICENSE)

build.py → build_cli.py

@@ -7,7 +7,7 @@
 
 def main():
     opts = [
-        'sv-dlp/__main__.py',
+        'sv_dlp/__main__.py',
         '--onefile',
         f'--name=sv-dlp{suffix}'
     ]

dev_scripts/cli_test.bat

@@ -0,0 +1,34 @@
+@REM genuinely the most rockiest programming i've done
+@echo OFF
+@echo Google Download
+py -Werror -Xdev -m "sv_dlp" 71xi-VgKXBBw5LvUWZMzfQ --service google
+py -Werror -Xdev -m "sv_dlp" 37.77499382574212, -122.47185699855395 --service google
+@echo Google Metadata Test
+py -Werror -Xdev -m "sv_dlp" 71xi-VgKXBBw5LvUWZMzfQ --get-m --service google
+py -Werror -Xdev -m "sv_dlp" 37.77499382574212, -122.47185699855395 --get-m --service google
+@echo Apple Download
+py -Werror -Xdev -m "sv_dlp" 37.77499382574212, -122.47185699855395 --service apple
+@echo Apple Metadata Test
+py -Werror -Xdev -m "sv_dlp" 37.77499382574212, -122.47185699855395 --get-m --service apple
+@echo Bing Download
+py -Werror -Xdev -m "sv_dlp" 37.77499382574212, -122.47185699855395 --service bing
+@echo Bing Metadata Test
+py -Werror -Xdev -m "sv_dlp" 37.77499382574212, -122.47185699855395 --get-m --service bing
+
+@REM Baidu Service Test
+@echo Baidu Download
+py -Werror -Xdev -m "sv_dlp" 39.90802391019931, 116.3403752455185 --service baidu
+@echo Baidu Metadata Test
+py -Werror -Xdev -m "sv_dlp" 39.90802391019931, 116.3403752455185 --get-m --service baidu
+
+@REM Yandex Service Test
+@echo Yandex Download
+py -Werror -Xdev -m "sv_dlp" 55.76550473786485, 37.54340745542864 --service yandex
+@echo Yandex Metadata Test
+py -Werror -Xdev -m "sv_dlp" 55.76550473786485, 37.54340745542864 --get-m --service yandex
+
+@REM Google Short Link Test
+@echo Google Short Link Test
+py -Werror -Xdev -m "sv_dlp" 71xi-VgKXBBw5LvUWZMzfQ -l
+py -Werror -Xdev -m "sv_dlp" 1.62097345657655 -75.61895756809007 -l
+py -Werror -Xdev -m "sv_dlp" https://goo.gl/maps/ebd71LHjZWF6HC3AA -p

dev_scripts/test.py

@@ -0,0 +1,84 @@
+"""
+Shall be ran under parent directory
+with python -m dev_scripts.test
+"""
+import sys
+import sv_dlp
+from pprint import pprint
+sv_dlp = sv_dlp.sv_dlp()
+
+def _pano_test(pano_id, service):
+    print(f"\n{service} panorama test")
+    sv_dlp.set_service(service)
+
+    md = sv_dlp.get_metadata(pano_id)
+    print(f"{md}\n{pano_id}")
+    print("Downloading...")
+    img = sv_dlp.download_panorama(pano_id=pano_id)
+    print("Saving...")
+    sv_dlp.postdownload.save_panorama(img, md)
+
+def _ll_test(lat, lng, service):
+    print(f"\n{service} download/metadata coordinate test")
+    sv_dlp.set_service(service)
+
+    md = sv_dlp.get_metadata(lat=lat, lng=lng)
+    pano = sv_dlp.get_pano_id(lat, lng)
+    print(f"{md}\n{pano}")
+    print("Downloading...")
+    img = sv_dlp.download_panorama(pano_id=pano)
+    print("Saving...")
+    sv_dlp.postdownload.save_panorama(img, md)
+
+if __name__ == "__main__":
+    available_services = sv_dlp.get_available_services()
+    print("Download/Metadata Test")
+    west_services = ["apple", "bing", "google"]
+
+    for service in available_services:
+        if service in west_services:
+            lat, lng = 37.77499382574212, -122.47185699855395
+            _ll_test(lat=lat, lng=lng, service=service)
+            if service == "google":
+                _pano_test(pano_id="YV7i9WYmvPqT5nEtFLq3SA", service=service)
+        elif service == "baidu":
+            lat, lng = 39.90802391019931, 116.3403752455185
+            pano_id = "09026600011611300908478628V"
+            _ll_test(lat=lat, lng=lng, service="baidu")
+            _pano_test(pano_id=pano_id, service="baidu")
+        elif "yandex":
+            lat, lng = 55.76550473786485, 37.54340745542864
+            pano_id = "1298034314_672832338_23_1528540830"
+            _pano_test(pano_id=pano_id, service="yandex")
+
+    print("\nURL <-> Coordinates Test")
+    for service in available_services:
+        match service:
+            case 'google':
+                print("\nGoogle URL test")
+                sv_dlp.set_service(service)
+                url = "https://goo.gl/maps/c1VXwkHcgj5d7sg16"
+                lat, lng = -11.66229059378587, -69.23418815832655
+            case 'baidu':
+                print("\nBaidu URL test")
+                sv_dlp.set_service(service)
+                url = "https://map.baidu.com/@11221564.09,2496329.11,21z,87t,-168.01h#panoid=09026600011611300908478628V&panotype=street&heading=76.14&pitch=1.38&l=21&tn=B_NORMAL_MAP&sc=0&newmap=1&shareurl=1&pid=09026600011611300908478628V"
+                lat, lng = 39.79019940979819, 116.34668908761401
+            case 'yandex':
+                print("\nYandex URL Test")
+                sv_dlp.set_service(service)
+                url = "https://yandex.com/maps/213/moscow/?l=stv%2Csta&ll=37.600082%2C55.790894&mode=search&panorama%5Bdirection%5D=286.651637%2C-5.390625&panorama%5Bfull%5D=true&panorama%5Bid%5D=1298034314_672832338_23_1528540830&panorama%5Bpoint%5D=37.599959%2C55.790842&panorama%5Bspan%5D=105.239256%2C60.000000&sll=37.600050%2C55.790854&text=55.790854%2C37.600050&z=20"
+                lat, lng = 41.3432002141999, 69.2646005452
+            case _:
+                continue
+
+        try:
+            pano = sv_dlp.get_pano_from_url(url)
+            print(pano)
+            url = sv_dlp.short_url(pano_id=pano)
+            print(url)
+            c_url = sv_dlp.short_url(lat=lat, lng=lng)
+            print(c_url)
+
+        except NameError: # if service not in match segment
+            continue