[WIP] Add generic GSoC 2017 import app #67

pradeepgangwar · 2017-12-22T07:01:30Z

currently work in progress.
Project rendering not implemented but rendering organization data implemented for reference and same can be done with projects once i get some reviews.
If YAML files are needed for reference i can add them here.

Please review and let me know changes it needs.

Related issue: #59

pradeepgangwar · 2017-12-22T07:15:30Z

Preview of what is extracted (org info)
https://deploy-preview-67--coala-community.netlify.com/gsoc/

jayvdb · 2018-01-03T01:43:37Z

gsoc/admin.py

@@ -0,0 +1,3 @@
+from django.contrib import admin


we cant use admin in a django-distill system

jayvdb · 2018-01-03T01:44:20Z

gsoc/config.py

+
+def load_cache(filename):
+    with open(os.path.join(DATA_DIR, filename), 'r') as f:
+        return ruamel.yaml.load(f, Loader=ruamel.yaml.Loader)


jayvdb · 2018-01-03T01:44:35Z

gsoc/config.py

+import ruamel.yaml
+import os
+
+DATA_DIR = os.path.join(


I think there is a global setting for this.

I didn't get this. I need more help here.

git grep _site

I searched but could not find any related setting other than STATIC_ROOT mentioned in community.settings as _site. I actually found this snippet of from gci app.

yes, that is it.

jayvdb · 2018-01-03T01:45:09Z

gsoc/tests.py

@@ -0,0 +1,3 @@
+from django.test import TestCase


delete this file until there are tests

jayvdb · 2018-01-03T01:50:28Z

templates/gsoc.html

+    <body>
+
+        {% if error %}
+            <h3> {{error}} </h3>


after the scraper is moved into django commands, the error log will appear at /log created in #63 , so you dont need to do error presentation here.

This is not actually error log. It appears in case when there are no data for a org. I am caching "no data present" also because it was stated in issue that we need to cache the fact that there is not data present for a org. Probably i should rename it? :).

It appears in case when there are no data for a org.

That is an error. It means the GSOC app needs to be disabled.

I have fixed this part. If related yaml file is not found then we show error. (file is not created if data is not found)

That is an error. It means the GSOC app needs to be disabled.

I do not delete existing yaml files on new query. Should i delete existing yaml files so that if query is made for another year(for which there is no data) so it does not render data from existing files. If there is no data no file gets created.

Remove this! Do not render errors here. This app should not exist if there was no gsoc data. it isnt a gsoc org. no app. no yaml. no page. using logging to record the gsoc app was disabled. no other visible trace.

jayvdb · 2018-01-03T02:43:55Z

gsocscrape/spiders/gsoc.py

+            technology['%s'% str(count)] = tech
+            count = count + 1
+
+        item =  {


double space.

jayvdb · 2018-01-03T02:45:14Z

gsocscrape/spiders/gsoc.py

+        project_data[int(item['id'])] = item
+        project_data = OrderedDict(sorted(project_data.items(), key=lambda t: t[0]))
+
+        with open(os.path.join('_site', 'project_info.yaml'), 'a') as f:


prefix fix with gsoc

jayvdb · 2018-01-03T02:46:07Z

gsocscrape/spiders/gsoc.py

+
+    def parse_project(self, response):
+        mentors = []
+        org_url = "https://summerofcode.withgoogle.com/archive/2017/organizations/"


year should be a parameter, as we'll also want to import other years, and other orgs will have many years of data to import.

jayvdb · 2018-01-03T02:46:37Z

gsocscrape/spiders/gsoc.py

+        yield response.follow(follow_link, self.parse_org)
+
+    def parse_org(self, response):        
+        project_url = "https://summerofcode.withgoogle.com/archive/2017/projects/"


year ; see below

jayvdb · 2018-01-03T02:49:20Z

gsoc/views.py

+            name = organisation[1].get('name')
+            return render(request, 'gsoc.html', {'error': name})
+
+        id = organisation[1].get('id')


why index of 1? that is a bit odd. the yaml should only have one org in it.

I got this running this way only. I think that's how it inteprets yaml. Although i will reconsider it and try it via key, value type object. Or it would be good if i get some suggestions :).

you dont need an array for one org. the Google json data structure that you fetch should not be the same as the coala org yaml data structure that you write to disk.

gitmate-bot · 2018-01-06T16:28:16Z

templates/gsoc_projects.html

+    </head>
+
+    <body>
+        <h3 style="text-align: center;"> Coala </h3>


The line contains the keyword 'Coala'.

Origin: KeywordBear, Section: generalization.

gitmate-bot · 2018-01-06T16:30:01Z

templates/gsoc_projects.html

+    </head>
+
+    <body>
+        <h3 style="text-align: center;"> coala </h3>


The line contains the keyword 'coala'.

Origin: KeywordBear, Section: generalization.

pradeepgangwar · 2018-01-06T16:51:50Z

Preview Links:
https://deploy-preview-67--coala-community.netlify.com/gsoc/projects
https://deploy-preview-67--coala-community.netlify.com/gsoc/

jayvdb · 2018-01-07T13:49:51Z

gsoc/config.py

+        return ruamel.yaml.load(f, Loader=ruamel.yaml.Loader)
+
+def get_year():
+    return "2017"


EOL at EOF.

but more importantly, returning 2017 here is inherently wrong. the year is now 2018. The GSoC 2018 program has now started.

And you want 2018 - 1

jayvdb · 2018-01-07T13:50:51Z

gsocscrape/items.py

@@ -0,0 +1,14 @@
+# -*- coding: utf-8 -*-
+
+# Define here the models for your scraped items


this is junk added by the generator to guide someone writing their code. It doesnt need to be in our repository

jayvdb · 2018-01-07T13:51:50Z

gsocscrape/spiders/gsoc.py

+class GsocSpider(scrapy.Spider):
+    name = "gsoc"
+    start_urls = [
+        "https://summerofcode.withgoogle.com/archive/{year}/organizations".format(year = year),


use pycodestyle on your code

jayvdb · 2018-01-10T08:45:10Z

gsoc/config.py

+import ruamel.yaml
+import os
+
+DATA_DIR = os.path.join(


yes, that is it.

jayvdb · 2018-01-10T08:45:18Z

gsoc/config.py

+
+DATA_DIR = os.path.join(
+    os.path.dirname(__file__), '..',
+    '_site'


Please use STATIC_ROOT here. (also create a newcomer issue to fix this in gci app)

jayvdb · 2018-01-10T08:46:39Z

gsoc/views.py

+            for technology in org.get(key).get('technologies').values():
+                tech.append(technology)
+
+            return render(request, 'gsoc.html', {'id': id, 'name': name,


move the substitution parameters to the next line

jayvdb · 2018-01-10T08:47:24Z

gsoc/views.py

+        logger.info('GSoC data not available')
+        error = 'No GSoC data available for {name} year {year}'.format(
+            name=org_name, year=year)
+        return render(request, 'gsoc.html', {'error': error})


no, stop rendering an error. use logging to record the error. and then re-raise the exception here.

jayvdb · 2018-01-10T08:48:12Z

gsoc/views.py

+            }
+            projects_list.append(item)
+        return render(request, 'gsoc_projects.html',
+                      {'project_list': projects_list, 'org_name': name})


lay out the these substitution variables so that you can add another one without modifying the existing lines.

jayvdb · 2018-01-10T08:51:10Z

templates/gsoc.html

+    <body>
+
+        {% if error %}
+            <h3> {{error}} </h3>


Remove this! Do not render errors here. This app should not exist if there was no gsoc data. it isnt a gsoc org. no app. no yaml. no page. using logging to record the gsoc app was disabled. no other visible trace.

gitmate-bot · 2018-01-14T16:09:39Z

gsocscrape/spiders/gsoc.py

+
+class GsocSpider(scrapy.Spider):
+    name = 'gsoc'
+    start_urls = [


You do not use the preferred quotation marks.

Origin: QuotesBear, Section: all.python.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpvhnur60c/gsocscrape/spiders/gsoc.py +++ b/tmp/tmpvhnur60c/gsocscrape/spiders/gsoc.py @@ -33,7 +33,7 @@ if(organization_link): organization_link = organization_link[0].extract().split('/')[4] else: - logger.info("Organisation {} does not exist in GSoC for {}".format( + logger.info('Organisation {} does not exist in GSoC for {}'.format( org_name, year )) return

gitmate-bot · 2018-01-14T16:09:40Z

gsoc/views.py

+            tagline = org.get(key).get('tagline')
+            description = org.get(key).get('description')
+            tech = []
+            for technology in org.get(key).get('technologies').values():


The code does not comply to PEP8.

Origin: PEP8Bear, Section: all.autopep8.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpvhnur60c/gsoc/views.py +++ b/tmp/tmpvhnur60c/gsoc/views.py @@ -33,7 +33,7 @@ 'tagline': tagline, 'description': description, 'tech': tech - }) + }) def projects(request):

gitmate-bot · 2018-01-14T16:09:41Z

gsoc/views.py

+        for key in org.keys():
+            name = org.get(key).get('name')
+        projects = get_projects_data()
+        projects_list = []


The code does not comply to PEP8.

Origin: PEP8Bear, Section: all.autopep8.

The issue can be fixed by applying the following patch:

--- a/tmp/tmpvhnur60c/gsoc/views.py +++ b/tmp/tmpvhnur60c/gsoc/views.py @@ -61,6 +61,6 @@ 'mentors': mentors } projects_list.append(item) - return render(request, 'gsoc_projects.html',{'project_list': projects_list, - 'org_name': name - }) + return render(request, 'gsoc_projects.html', {'project_list': projects_list, + 'org_name': name + })

gitmate-bot · 2018-01-14T16:09:42Z

gsoc/views.py

+            tagline = org.get(key).get('tagline')
+            description = org.get(key).get('description')
+            tech = []
+            for technology in org.get(key).get('technologies').values():


E124 closing bracket does not match visual indentation

Origin: PycodestyleBear (E124), Section: all.autopep8.

gitmate-bot · 2018-01-14T16:09:42Z

gsoc/views.py

+        for key in org.keys():
+            name = org.get(key).get('name')
+        projects = get_projects_data()
+        projects_list = []


E231 missing whitespace after ','

Origin: PycodestyleBear (E231), Section: all.autopep8.

gitmate-bot · 2018-01-14T16:09:43Z

gsoc/views.py

+        for key in org.keys():
+            name = org.get(key).get('name')
+        projects = get_projects_data()
+        projects_list = []


E501 line too long (83 > 80 characters)

Origin: PycodestyleBear (E501), Section: all.autopep8.

gitmate-bot · 2018-01-14T16:09:44Z

gsoc/views.py

+        projects = get_projects_data()
+        projects_list = []
+        for key in projects.keys():
+            mentors = []


E124 closing bracket does not match visual indentation

Origin: PycodestyleBear (E124), Section: all.autopep8.

gitmate-bot · 2018-01-14T16:09:45Z

gsoc/views.py

+        for key in org.keys():
+            name = org.get(key).get('name')
+        projects = get_projects_data()
+        projects_list = []


Line is longer than allowed. (83 > 80)

Origin: LineLengthBear, Section: all.linelength.

pradeepgangwar · 2018-01-14T16:56:20Z

No file gets created if there is no data. I raised 404 exception as i didn't get any way to disable an app. Is there any way to disable the app for a condition.

jayvdb · 2018-01-28T00:59:36Z

raising 404 is good enough.

Note that there is no problem with the GCI app. master builds on cron work every night. see https://travis-ci.org/coala/community/builds . Something is wrong in your changes.

pradeepgangwar · 2018-02-02T06:52:14Z

@jayvdb I will update it. Thanks.

Closes #59

pradeepgangwar · 2018-02-05T09:06:08Z

https://deploy-preview-67--coala-community.netlify.com/gsoc/projects
https://deploy-preview-67--coala-community.netlify.com/gsoc

RaiVaibhav · 2018-08-24T18:57:28Z

@pradeepgangwar @jayvdb I tried to clone the forked repo and tried to run the .ci scripts so that I can see the data locally but an error occurred
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pradeepgangwar.github.io/community/static/tasks.yaml

Please help me in setting up.

RaiVaibhav · 2018-08-24T18:58:20Z

Please tell me if I am missing some step

pradeepgangwar · 2018-08-29T09:39:34Z

Closing this due to inactivity. I am not able to take it to completion.

jayvdb requested changes Jan 3, 2018

View reviewed changes

jayvdb requested a review from mamat-rahmat January 6, 2018 04:57

gitmate-bot added process/wip size/M labels Jan 6, 2018

gitmate-bot reviewed Jan 6, 2018

View reviewed changes

jayvdb reviewed Jan 7, 2018

View reviewed changes

coala deleted a comment from gitmate-bot Jan 10, 2018

jayvdb requested changes Jan 10, 2018

View reviewed changes

gitmate-bot reviewed Jan 14, 2018

View reviewed changes

gitmate-bot added the status/STALE label Jan 22, 2018

gitmate-bot removed the status/STALE label Jan 28, 2018

pradeepgangwar closed this Feb 4, 2018

pradeepgangwar mentioned this pull request Feb 4, 2018

Add generic GSoC import app #76

Closed

pradeepgangwar reopened this Feb 5, 2018

gitmate-bot added initiatives/gsoc size/L and removed size/M labels Feb 5, 2018

Add generic gsoc import app

0671cf3

Closes #59

gitmate-bot added size/M and removed size/L labels Feb 5, 2018

gitmate-bot added the status/STALE label Feb 13, 2018

gitmate-bot removed the status/STALE label Aug 24, 2018

pradeepgangwar closed this Aug 29, 2018

pradeepgangwar deleted the gsoc-import branch August 29, 2018 09:39

		@@ -0,0 +1,14 @@
		# -- coding: utf-8 --

		# Define here the models for your scraped items

[WIP] Add generic GSoC 2017 import app #67

[WIP] Add generic GSoC 2017 import app #67

Conversation

pradeepgangwar commented Dec 22, 2017 • edited Loading

pradeepgangwar commented Dec 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradeepgangwar Jan 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayvdb Jan 7, 2018 • edited Loading

Choose a reason for hiding this comment

pradeepgangwar Jan 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradeepgangwar Jan 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradeepgangwar commented Jan 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pradeepgangwar commented Jan 14, 2018 • edited Loading

jayvdb commented Jan 28, 2018

pradeepgangwar commented Feb 2, 2018

pradeepgangwar commented Feb 5, 2018

RaiVaibhav commented Aug 24, 2018

RaiVaibhav commented Aug 24, 2018

pradeepgangwar commented Aug 29, 2018

pradeepgangwar commented Dec 22, 2017 •

edited

Loading

pradeepgangwar commented Dec 22, 2017 •

edited

Loading

pradeepgangwar Jan 7, 2018 •

edited

Loading

jayvdb Jan 7, 2018 •

edited

Loading

pradeepgangwar Jan 7, 2018 •

edited

Loading

pradeepgangwar Jan 6, 2018 •

edited

Loading

pradeepgangwar commented Jan 14, 2018 •

edited

Loading